From rousskov at measurement-factory.com Fri Mar 4 18:19:18 2011 From: rousskov at measurement-factory.com (Alex Rousskov) Date: Fri, 04 Mar 2011 11:19:18 -0700 Subject: Polygraph v4.3.1 available Message-ID: <4D712D26.1050701@measurement-factory.com> Hello, Web Polygraph v4.3.1 is now available at http://www.web-polygraph.org/downloads/ Compared to the last public release (v4.0.11), this version includes many new features and improvements. Here is a brief summary: - SOCKS support. - Content-driven robots. - Transaction lifetime limits. - PGL phase script access to live transaction stats. - Better randomness and consistency in traffic generation models. - Better support for trace replay. - PGL-configurable custom transaction statistics. - IPv6, DNS, and HTTP support fixes. - Portability improvements and bug fixes. Please see the change log quoted below for details. Enjoy, Alex. ------------ Change Log since v4.0.11 --------------- > version 4.3.1 > 2011/03/03 > > - Better support for trace replay: Ignore Robot.origins field > for PGL Robots with foreign-only interest. Before the change, > non-empty Robot.origins field was required, even if it was > unused because the Robot has no interest other than replaying > the foreign trace. > > - Better support for trace replay: Do not "warm up" Robots with > foreign-only interest. Before the change, many workloads > containing trace-replaying Robots could get stuck in the first > phase due to an infinite warmup. Now, foreign-only Robots do > not affect the warmup, and the first phase ends as expected. > > - Print the corresponding workload file line and number for many > PGL-related warnings and errors. > > - Fixed client crash on exit when Robot.accept_content_encodings > PGL field is set. > > - Fixed generation of lifecycle-related entity parameters when > the same ObjLifeCycle object is used multiple times in the > workload. The bug resulted in unexpected entity-header (e.g., > Last-Modified-Time and Expires) changes when the same URL was > revisited. > > - Fix statistics recording for aborted transactions. Before the > change, non-basic (e.g. POST) aborted transactions were > recorded in the wrong category (e.g. post). Now, all aborted > transactions are recorded in the aborted category. > > - OpenSSL 1.0 support (LP:690599). > > - Fixed epoll(2)-related bug which could result in "Socket not > connected" and similar errors. > > - Fixed select(2) support in SSL-related workloads. > > - Code cleanup. > > > version 4.3.0 > 2010/12/10 > > - Content-driven Robots support: The user may configure Robot > behavior based on Content. Real HTTP client behavior often > depends on web site and its content. Content-driven Robots > help simulate those dependencies. > > All for-Robot settings in PGL Content objects are grouped > under the new Content::client_behavior field. A new PGL > ClientBehavior type is added. Assigning PGL Robot to > ClientBehavior and vice versa is supported. Once Server and > Content are selected, Content::client_behavior settings, if > any, overwrite the corresponding Robot fields. > > For now, the following three Robot behavior properties can be > specified on per-Content basis: req_types, req_methods, and > ranges. > > - Considerably improved the randomness of distribution values > tied to URLs, such as response sizes. The old distributions > could have too many similar values, with too few extreme > values, especially when non-unique worlds (--unique_world no) > were used with multiple PGL Servers and table-based > distributions. > > - Linked object blobs from traffic stream tables in report. > > - Fixed poll(2)-related bug which could result in "Socket not > connected" or similar errors on platforms without epoll(2) > support such as Mac OS X. > > - Multiple world support for distr_test tool: The world_count > option is intended to simulate URL-based value generation in > workloads with multiple PGL Servers, each Server being a "oid > world" for the Robot. By default, world_count is 1, and all > oids are generated from a single world. If world_count is > greater than 1, distr_test generates oids from multiple > worlds. The number of oids generated from each world is > (trials count / worlds count). > > - Minor fixes and code cleanup. > > > version 4.2.2 > 2010/11/05 > > - Added custom stats: The user may configure which transactions > are added to custom stats based on HTTP status code. > Configuration is done with Agent.custom_stats_scope PGL field: > > custom_stats_scope = ["all", "-2xx", "+30x", "+206"]; > > If a transaction finishes before getting a reply, or there is > no HTTP status code for another reason, the transaction is not > included in the custom stats. > > The following custom stats are collected: custom.rep, > custom.live.rep, custom.projected.rep. Custom stats are > accessible from PGL phase scripts. > > - Fixed PGL TblDistr to produce the same results for the same > seed. For example, if a table distribution is used for > response body sizes, the response size would stay the same > when the same URL is visited multiple times. > > - Fixed several build and startup problems on Mac OS X. > > - Fixed a DummyNet-related build problem on FreeBSD 8.1. > > - Fixed IPv6 address parsing in command line arguments. > > - Fixed segfault when parsing a PGL array or list with a > malformed item. > > - Allow IPv6 addresses in PGL to start with '[:'. > > > version 4.2.1 > 2010/10/14 > > - Transaction lifetime support: Allow the user to limit the > lifetime of client and server transactions via the new PGL > Agent::xact_lifetime field. A transaction aborts when its > configured lifetime expires. Aborted transactions are treated > as errors on client and server sides of the test, for now. > > Client transaction lifetime includes both connection slot > waiting time (if any; see Robot::open_conn_lmt) and regular > execution time. Server transaction lifetime includes both think > time (if any; see Agent::xact_think) and regular execution time. > > > version 4.2.0 > 2010/10/08 > > - Allow PGL scripts to monitor in-flight or "live" transactions > so that they can adjust load even when many transactions get > stuck or slow down and do not immediately contribute to regular > postmortem stats. Measure how long live transactions have > been running (live.rep.rptm) and how many response bytes they > have read (live.rep.size). Make these new stats available to > PGL scripts via the regular stats interfaces > (currentSample().live.* and currentPhase().live.*). > > - Added projected.rep.* statistics that combines live and > finished transaction stats. Available to PGL scripts only. > > - Treat an HTTP CONNECT response as the end of the transaction. > Collect and report stats dedicated to these new CONNECT-only > transactions. These CONNECT transactions are often rather > different from the in-tunnel transactions that follow. > Isolating them also prevents CONNECT overheads from polluting > the first in-tunnel transaction stats. > > > version 4.1.2 > 2010/11/06 > > - Fixed PGL TblDistr to produce the same results for the same > seed. For example, if a table distribution is used for > response body sizes, the response size would stay the same > when the same URL is visited multiple times. > > - Fixed several build and startup problems on Mac OS X. > > - Fixed a DummyNet-related build problem on FreeBSD 8.1. > > - Fixed IPv6 address parsing in command line arguments. > > - Fixed segfault when parsing a PGL array or list with a > malformed item. > > - Allow IPv6 addresses in PGL to start with '[:'. > > > version 4.1.1 > 2010/10/07 > > - Add dump command to the cdb tool, which stores all verbatim and > link-only objects into individual files. Cdb dump accepts an > optional --as "path/prefix${name}.suffix " parameter. Default > --as value is "./cdb${name}". ${name} macros are substituted with > 1,2,3,4... integers. > > - Cdb interface has changed. Commands (add, dump, show) are passed > as options with database argument, i.e. cdb --show test.cdb. > > - Limit PGL load and population factors value to [0%, 100%] range. > Print error and exit if begin or end factor value is greater than > 100%. If a script changes factor to a negative or greater than > 100% value, print warning and set the factor to 0%/100% instead. > > - Set Recursion Desired (RD) flag for outgoing DNS queries because > some modern resolvers do not accept queries without that flag > and because we do not recursively resolve ourselves. > > - Skip already configured network interface aliases when creating > new ones if delete_old_addrs option is false. > > - Do not send Content-MD5 header with HTTP 206 (Partial Content) > responses. This fixes segfault and is in line with HTTPbis. > See http://trac.tools.ietf.org/wg/httpbis/trac/ticket/178 > > - Fix assertion during network interface alias creation. > > - Fix errPersistButNoCLen error on client-side when multi-ranges > and persistent connections are enabled. Content-Length is not > required if message uses the media type "multipart/byteranges". > > - Fix LP:644813: SSL issues with DNS and embedded objects. > > - Fix address space size check for SpreadAs. > > - Fix multiple alias range support in the aka tool. > > - Fix compilation of the msl_test tool on Linux. > > - Make out-of-memory handler more robust. > > - Minor fixes and code cleanup. > > > version 4.1.0 > 2010/05/13 > > - Support SOCKS5 proxies for HTTP, HTTPS, and passive FTP > traffic (socks_proxies and socks_prob PGL Robot fields). No > support for Active FTP data connections yet. "None" and basic > "username/password" SOCKS authentication methods are > supported. Chaining SOCKS and HTTP proxies is supported (the > socks_chaining_prob PGL Robot field). > > - Extend the ipsToNames() PGL function to support a domain name > generation template with ${dashed_ip} and ${port} macros. This > change allows more flexibility in naming a large number of > server domains. > > - Add a dynamize() PGL function that converts an array of static > domain names to dynamic names (using the supplied renewal > percentage), similar to what the dynamicName() function does > to one static name. This change allows creation of a large > number of servers, each with a unique dynamic domain name. > > - Do not send "100 Continue" control response if we got the > entire request body already. This fix prevents POST requests > sent without waiting for 100 Continue response from getting > stuck because the client sent everything and cannot continue. > > - Fix server freeze when "Expect: 100-continue" request headers > and body (partial or complete) is received in a single read(2) > call. > > - Fix assertion when an HTTP client aborts transaction. > > - Fix abort position calculation on the client side. The abort > size was set bigger than the request size, and client aborted > after the request was sent, during response reading. > From fclaire at free.fr Wed Mar 9 06:07:09 2011 From: fclaire at free.fr (Francois Claire) Date: Wed, 09 Mar 2011 07:07:09 +0100 Subject: Polygraph v4.3.1 available In-Reply-To: <4D712D26.1050701@measurement-factory.com> References: <4D712D26.1050701@measurement-factory.com> Message-ID: <4D77190D.8090509@free.fr> Thanks a lot for your work and congratulations for this new release! I'm also pleased to announce Polygraph 4.3.1 is now available for all Mac OS X systems through the macports.org project: http://www.macports.org/ports.php?by=name&substr=polygraph Regards, Francois. Le 04/03/11 19:19, Alex Rousskov a ?crit : > Hello, > > Web Polygraph v4.3.1 is now available at > http://www.web-polygraph.org/downloads/ > > Compared to the last public release (v4.0.11), this version includes > many new features and improvements. Here is a brief summary: > > - SOCKS support. > - Content-driven robots. > - Transaction lifetime limits. > - PGL phase script access to live transaction stats. > - Better randomness and consistency in traffic generation models. > - Better support for trace replay. > - PGL-configurable custom transaction statistics. > - IPv6, DNS, and HTTP support fixes. > - Portability improvements and bug fixes. > > Please see the change log quoted below for details. > > > Enjoy, > > Alex. > From pmix at hendrie.id.au Tue Mar 15 06:10:32 2011 From: pmix at hendrie.id.au (Michael Hendrie) Date: Tue, 15 Mar 2011 16:40:32 +1030 Subject: DNS Config Tool with Mulitple Address Maps Message-ID: <3A173D57-DCE4-4A37-A96D-60A4F962C09C@hendrie.id.au> Hello List, I am attempting to use a single configuration file to run a Polygraph workload that contains both normal HTTP and SSL robots/servers. This appears to be working well, when I start the server process I see it creates the HTTP Server agents listening on port 80 and the SSL server agents listening on port 443. I have started the client process and this loads the configuration file without complaining about errors. The issue is generating the DNS configuration file for BIND. When I run the polygraph-dns-cfg tool it errors out complaining about multiple address maps being used. My understanding from http://www.web-polygraph.org/docs/reference/pgl/types.html#type:docs/reference/pgl/types/AddrMap is that multiple address maps should be able to be used. The errors is: bash-3.2# /usr/local/bin/polygraph-dns-cfg --config ../mymixed.pg ../mymixed.pg: must use() exactly one address map, but 2 maps were used I have tested with both web-polygraph 4.0.10 and 4.3.1 but both produce the same error. The relevant part of the workload file is: AddrMap SslMap = { zone = "sslbench.tst"; addresses = srvSsl.addresses; names = ipsToNames(addresses, zone); }; AddrMap HttpMap = { zone = "httpbench.tst"; addresses = srvHttp.addresses; names = ipsToNames(addresses, zone); }; ... ... ... use(SslMap, HttpMap); Does anyone know how to overcome this issue and generate a useful bind file with the tool when multiple address maps are used. Thanks Mick From dmitry.kurochkin at measurement-factory.com Tue Mar 15 21:21:04 2011 From: dmitry.kurochkin at measurement-factory.com (Dmitry Kurochkin) Date: Wed, 16 Mar 2011 00:21:04 +0300 Subject: DNS Config Tool with Mulitple Address Maps In-Reply-To: <3A173D57-DCE4-4A37-A96D-60A4F962C09C@hendrie.id.au> References: <3A173D57-DCE4-4A37-A96D-60A4F962C09C@hendrie.id.au> Message-ID: <87d3lsdqrj.fsf@gmail.com> Hi Michael. On Tue, 15 Mar 2011 16:40:32 +1030, Michael Hendrie wrote: > Hello List, > > I am attempting to use a single configuration file to run a Polygraph workload that contains both normal HTTP and SSL robots/servers. This appears to be working well, when I start the server process I see it creates the HTTP Server agents listening on port 80 and the SSL server agents listening on port 443. I have started the client process and this loads the configuration file without complaining about errors. > > The issue is generating the DNS configuration file for BIND. When I run the polygraph-dns-cfg tool it errors out complaining about multiple address maps being used. My understanding from http://www.web-polygraph.org/docs/reference/pgl/types.html#type:docs/reference/pgl/types/AddrMap is that multiple address maps should be able to be used. > > The errors is: > > bash-3.2# /usr/local/bin/polygraph-dns-cfg --config ../mymixed.pg > ../mymixed.pg: must use() exactly one address map, but 2 maps were used > Currently, this is a limitation of dns-cfg tool. Polygraph client supports multiple address maps but the DNS configuration tool does not. As a workaround, you may use dns-cfg multiple times use()ing different address maps in PGL workload and merge output by hand. Note: Only BIND configuration (i.e. dns-cfg console output) needs merging, individual zone files (e.g. tst and tst.rev) do not need any changes. Added to FAQ on Launchpad [1]. Regards, Dmitry [1] https://answers.launchpad.net/polygraph/+faq/1512 > > I have tested with both web-polygraph 4.0.10 and 4.3.1 but both produce the same error. > > The relevant part of the workload file is: > > AddrMap SslMap = { > zone = "sslbench.tst"; > addresses = srvSsl.addresses; > names = ipsToNames(addresses, zone); > }; > > AddrMap HttpMap = { > zone = "httpbench.tst"; > addresses = srvHttp.addresses; > names = ipsToNames(addresses, zone); > }; > > ... > ... > ... > use(SslMap, HttpMap); > > Does anyone know how to overcome this issue and generate a useful bind file with the tool when multiple address maps are used. > > Thanks > > Mick > > _______________________________________________ > Users mailing list > Users at web-polygraph.org > http://www.web-polygraph.org/mailman/listinfo/users From huaraz at moeller.plus.com Fri Mar 18 19:25:59 2011 From: huaraz at moeller.plus.com (Markus Moeller) Date: Fri, 18 Mar 2011 19:25:59 -0000 Subject: Configuration with Negotiate/NTLM Message-ID: I would like to test NTLM authentication but my (very) simple config does not work when the proxy asks for Negotiate. I see an NTLM token going to the proxy but with DOMAIN NULL. I also don't see any traffic from the client nor proxy to AD. I know my client can talk to AD ( at least the winbind check work). ... Robot R3 = { kind = "R101"; pop_model = { pop_distr = popUnif(); }; recurrence = 55% / SimpleContent.cachable; // adjusted to get 55% DHR origins = S.addresses; // where the origin servers are addresses = [ '192.168.1.127' ]; // where these robot agents will be created credentials = [ "WIN2003R2/mm at client1:pwd" ]; dns_resolver = dr; pconn_use_lmt = exp(10*1024); }; ... Do I have to configure more ? Thank you Markus From dmitry.kurochkin at measurement-factory.com Wed Mar 23 18:17:31 2011 From: dmitry.kurochkin at measurement-factory.com (Dmitry Kurochkin) Date: Wed, 23 Mar 2011 21:17:31 +0300 Subject: Configuration with Negotiate/NTLM In-Reply-To: References: Message-ID: <87vcz9ln0k.fsf@gmail.com> Hi Markus. It looks like you have configured everything correctly. Setting credentials (POLYDOMAIN/polyuser at polyhost:polypass) and pconn_use_lmt should be enough. I have tested NTLM authentication with Web Polygraph v4.3.1 and it works fine: * client sends request to the proxy with no authentication headers * server replies with 407 (Proxy Authentication Required) and "Proxy-Authenticate: NTLM" header * client sends request with NTLMSSP_NEGOTIATE message with null workstation name and null workstation name * server replies with 407 (Proxy Authentication Required) and NTLMSSP_CHALLENGE message * client sends request on the same TCP connection with NTLMSSP_AUTH message * server replies with 200 (OK) and no authentication headers The important part is that client sends requests on the same TCP connection, that is why pconn_use_lmt is needed. It seems that it is expected that client sends the first NTLM token with null domain. Only the second client request (after the proxy replies with NTLMSSP_CHALLENGE) contains domain, host and username. Client does not communicate with anything but the proxy. I guess in your setup the proxy would talk to AD. According to your further findings [1], this might be a Squid bug. Let's see what it boils over to. Regards, Dmitry [1] http://thread.gmane.org/gmane.comp.web.squid.devel/14886 From dane.elwell at gmail.com Wed Mar 30 20:33:04 2011 From: dane.elwell at gmail.com (Dane Elwell) Date: Wed, 30 Mar 2011 21:33:04 +0100 Subject: polygraph-lx --objects Message-ID: Hello, I'm rather embarrassed to be asking this question, but my Google-fu has reached its limit. Is there a list of the objects I am able to extract with the --objects command line option, along with a description of that object, available anywhere? It seems I am unable to decipher the meaning of 'rep.rptm.mean', amongst others. I cannot find a description of these objects anywhere. Thanks Dane