From g.hohl at aurenz.de Thu Dec 1 08:24:37 2011 From: g.hohl at aurenz.de (Hohl, Gerrit) Date: Thu, 1 Dec 2011 09:24:37 +0100 Subject: AW: polygraph-pgl2acl doesn't recognize credentials In-Reply-To: <4ED658F3.70007@measurement-factory.com> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8A457C@srvmail2.aurenz.local> <4ED658F3.70007@measurement-factory.com> Message-ID: <3B7A7BDFFD286F49A19A5B35D26E08FB8A45C2@srvmail2.aurenz.local> Hello Alex, thanks for the fast reply. :-) I'm a little bit confused as there is no ACL property in the robot type: http://www.web-polygraph.org/docs/reference/pgl/types.html#type:docs/reference/pgl/types/Robot By the way: There is also no credentials property or even a credentials(int, String) call in the documentation. ;-) Hm, I guess in this case it is easier for me to write a little program that creates a file which I can include in the proxy. That program will create another pgl file which I can include in the test case and use the array in the robot. In the case of using squid with ncsa_auth it would mean that it would produce a file (in this example "cred.store") in htpasswd style (by calling the htpasswd command): bob:$apr1$fE4.....$qjlBH02N80pZieEvIeQ73/ mary:$apr1$BG4.....$KnQqqBZXre.VyVOfaAls20 duane:$apr1$/H4.....$uZSzXgJYuLLf/JKVkPOOM0 Then I will modify the /etc/squid.conf and add / modify the following lines: auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/cred.store auth_param basic children 5 acl authuser proxy_auth REQUIRED #http_access allow localnet #http_access allow localhost http_access allow authusers The pg file (in this case "include/cred.pg") would look like that: string[] cred = [ "bob:secret", "mary:letMeIn", "duane:p$987<4" ]; In the include/polymix-4-guts.pg I will add: #include "cred.pg" and modify the robot by adding the following lines: auth_error = 0.1% credentials = cred; That should work, shouldn't it? Regards Gerrit -----Urspr?ngliche Nachricht----- Von: Alex Rousskov [mailto:rousskov at measurement-factory.com] Gesendet: Mittwoch, 30. November 2011 17:25 An: Hohl, Gerrit Cc: users at web-polygraph.org Betreff: Re: polygraph-pgl2acl doesn't recognize credentials On 11/30/2011 07:36 AM, Hohl, Gerrit wrote: > my project manager introduced web-polygraph to me. We want to test our > own proxy with it. Hello Gerrit, Welcome to the list. > I'm trying to include authentication into the test. I added the > following line in the robot: > > credentials = credentials(count(addresses), "authmix"); > > and started polygraph-pgl2acl. But it only shows that it couldn't find > any authentication rules: > > polygraph-pgl2acl: parsing... > polygraph-pgl2acl: collecting rules... > polygraph-pgl2acl: pruning 0 rules ... > polygraph-pgl2acl: sorting 0 rules ... > polygraph-pgl2acl: symplifying... > polygraph-pgl2acl: printing 0 rules ... > > Normally there should be around 125 IP adresses for the robots. So > shouldn't there be also 125 credentials? Why doesn't it find anything? > I don't know what I did wrong. Maybe one of you can help me finding my > mistake. The pgl2acl tool produces Access Control Lists (ACLs) from Robot.acl fields and is not directly related to Robot IP addresses or Robot credentials. I suspect your PGL Robots had no acl rules so the tool produced nothing. You probably do not need those rules if all you are interested in is authentication. What you probably want is the polygraph-pgl2ldif tool. See http://www.web-polygraph.org/docs/userman/auth.html#Sect:3.2 and search for "human-friendly" to find a non-LDIF example that may be useful for generating rules to configure your proxy. HTH, Alex. From rousskov at measurement-factory.com Thu Dec 1 16:09:43 2011 From: rousskov at measurement-factory.com (Alex Rousskov) Date: Thu, 01 Dec 2011 09:09:43 -0700 Subject: AW: polygraph-pgl2acl doesn't recognize credentials In-Reply-To: <3B7A7BDFFD286F49A19A5B35D26E08FB8A45C2@srvmail2.aurenz.local> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8A457C@srvmail2.aurenz.local> <4ED658F3.70007@measurement-factory.com> <3B7A7BDFFD286F49A19A5B35D26E08FB8A45C2@srvmail2.aurenz.local> Message-ID: <4ED7A6C7.3010809@measurement-factory.com> On 12/01/2011 01:24 AM, Hohl, Gerrit wrote: > I'm a little bit confused as there is no ACL property in the robot type: > http://www.web-polygraph.org/docs/reference/pgl/types.html#type:docs/reference/pgl/types/Robot > By the way: There is also no credentials property or even a credentials(int, String) call in the documentation. ;-) Sorry, some Polygraph features are not yet fully documented. Feel free to file a bug report for the ones you noticed -- it will help developers to keep track of what they need to fix: https://bugs.launchpad.net/polygraph > Hm, I guess in this case it is easier for me to write a little > program that creates a file which I can include in the proxy. That > program will create another pgl file which I can include in the test > case and use the array in the robot. Sure, that approach should work. Please also consider adding a DES(?) wrapping function to the pgl2ldif program instead. It would be a generally useful addition that allows anybody to easily produce htpasswd-style files from PGL workloads. Cheers, Alex. From g.hohl at aurenz.de Fri Dec 2 08:08:29 2011 From: g.hohl at aurenz.de (Hohl, Gerrit) Date: Fri, 2 Dec 2011 09:08:29 +0100 Subject: AW: AW: polygraph-pgl2acl doesn't recognize credentials In-Reply-To: <4ED7A6C7.3010809@measurement-factory.com> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8A457C@srvmail2.aurenz.local> <4ED658F3.70007@measurement-factory.com> <3B7A7BDFFD286F49A19A5B35D26E08FB8A45C2@srvmail2.aurenz.local> <4ED7A6C7.3010809@measurement-factory.com> Message-ID: <3B7A7BDFFD286F49A19A5B35D26E08FB8F102D@srvmail2.aurenz.local> Hello Alex, > Sorry, some Polygraph features are not yet fully documented. Feel free > to file a bug report for the ones you noticed -- it will help developers > to keep track of what they need to fix: > https://bugs.launchpad.net/polygraph okay, I will do that. :-) Also I'm not sure in all cases if it is a problem of the documentation or if it depends on other things (e.g. in the Ubuntu polygraph package the names of the executables are different and polypxy doesn't exist). > Sure, that approach should work. Please also consider adding a DES(?) > wrapping function to the pgl2ldif program instead. It would be a > generally useful addition that allows anybody to easily produce > htpasswd-style files from PGL workloads. I have to admit that I wrote a small Java program to solve that problem instead of modifying the C code of web polygraph. It uses the htpasswd executable (from the apache-util package) for creating the apache-style password files. Regards, Gerrit -----Urspr?ngliche Nachricht----- Von: Alex Rousskov [mailto:rousskov at measurement-factory.com] Gesendet: Donnerstag, 1. Dezember 2011 17:10 An: Hohl, Gerrit Cc: users at web-polygraph.org Betreff: Re: AW: polygraph-pgl2acl doesn't recognize credentials On 12/01/2011 01:24 AM, Hohl, Gerrit wrote: > I'm a little bit confused as there is no ACL property in the robot type: > http://www.web-polygraph.org/docs/reference/pgl/types.html#type:docs/r > eference/pgl/types/Robot By the way: There is also no credentials > property or even a credentials(int, String) call in the documentation. > ;-) Sorry, some Polygraph features are not yet fully documented. Feel free to file a bug report for the ones you noticed -- it will help developers to keep track of what they need to fix: https://bugs.launchpad.net/polygraph > Hm, I guess in this case it is easier for me to write a little program > that creates a file which I can include in the proxy. That program > will create another pgl file which I can include in the test case and > use the array in the robot. Sure, that approach should work. Please also consider adding a DES(?) wrapping function to the pgl2ldif program instead. It would be a generally useful addition that allows anybody to easily produce htpasswd-style files from PGL workloads. Cheers, Alex. From dmitry.kurochkin at measurement-factory.com Fri Dec 2 08:37:20 2011 From: dmitry.kurochkin at measurement-factory.com (Dmitry Kurochkin) Date: Fri, 02 Dec 2011 12:37:20 +0400 Subject: AW: AW: polygraph-pgl2acl doesn't recognize credentials In-Reply-To: <3B7A7BDFFD286F49A19A5B35D26E08FB8F102D@srvmail2.aurenz.local> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8A457C@srvmail2.aurenz.local> <4ED658F3.70007@measurement-factory.com> <3B7A7BDFFD286F49A19A5B35D26E08FB8A45C2@srvmail2.aurenz.local> <4ED7A6C7.3010809@measurement-factory.com> <3B7A7BDFFD286F49A19A5B35D26E08FB8F102D@srvmail2.aurenz.local> Message-ID: <87k46fwerj.fsf@gmail.com> Hi Gerrit. On Fri, 2 Dec 2011 09:08:29 +0100, "Hohl, Gerrit" wrote: > Hello Alex, > > > Sorry, some Polygraph features are not yet fully documented. Feel free > > to file a bug report for the ones you noticed -- it will help developers > > to keep track of what they need to fix: > > https://bugs.launchpad.net/polygraph > > okay, I will do that. :-) > Also I'm not sure in all cases if it is a problem of the documentation > or if it depends on other things (e.g. in the Ubuntu polygraph package > the names of the executables are different and polypxy doesn't exist). > It is outdated documentation. I maintain Debian (and Ubuntu) Web Polygraph package. The package does not do any renames or other weird stuff. You would get the same executable names if you compile by hand and do make install. In Web Polygraph v4.0.8 many executables were renamed and "polygraph-" prefix is now added during install. In particular, polyclt and polysrv are polygraph-client and polygraph-server now. Unfortunately, documentation is not yet fully updated. As for polypxy tool, it has not been available for a long time. Again, documentation is not up-to-date here. Regards, Dmitry > > Sure, that approach should work. Please also consider adding a DES(?) > > wrapping function to the pgl2ldif program instead. It would be a > > generally useful addition that allows anybody to easily produce > > htpasswd-style files from PGL workloads. > > I have to admit that I wrote a small Java program to solve that problem instead of modifying the C code of web polygraph. It uses the htpasswd executable (from the apache-util package) for creating the apache-style password files. > > > > Regards, > Gerrit > > -----Urspr?ngliche Nachricht----- > Von: Alex Rousskov [mailto:rousskov at measurement-factory.com] > Gesendet: Donnerstag, 1. Dezember 2011 17:10 > An: Hohl, Gerrit > Cc: users at web-polygraph.org > Betreff: Re: AW: polygraph-pgl2acl doesn't recognize credentials > > On 12/01/2011 01:24 AM, Hohl, Gerrit wrote: > > > I'm a little bit confused as there is no ACL property in the robot type: > > http://www.web-polygraph.org/docs/reference/pgl/types.html#type:docs/r > > eference/pgl/types/Robot By the way: There is also no credentials > > property or even a credentials(int, String) call in the documentation. > > ;-) > > Sorry, some Polygraph features are not yet fully documented. Feel free to file a bug report for the ones you noticed -- it will help developers to keep track of what they need to fix: > https://bugs.launchpad.net/polygraph > > > > Hm, I guess in this case it is easier for me to write a little program > > that creates a file which I can include in the proxy. That program > > will create another pgl file which I can include in the test case and > > use the array in the robot. > > Sure, that approach should work. Please also consider adding a DES(?) wrapping function to the pgl2ldif program instead. It would be a generally useful addition that allows anybody to easily produce htpasswd-style files from PGL workloads. > > > Cheers, > > Alex. > _______________________________________________ > Users mailing list > Users at web-polygraph.org > http://www.web-polygraph.org/mailman/listinfo/users From g.hohl at aurenz.de Fri Dec 2 08:45:23 2011 From: g.hohl at aurenz.de (Hohl, Gerrit) Date: Fri, 2 Dec 2011 09:45:23 +0100 Subject: AW: AW: AW: polygraph-pgl2acl doesn't recognize credentials In-Reply-To: <87k46fwerj.fsf@gmail.com> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8A457C@srvmail2.aurenz.local> <4ED658F3.70007@measurement-factory.com> <3B7A7BDFFD286F49A19A5B35D26E08FB8A45C2@srvmail2.aurenz.local> <4ED7A6C7.3010809@measurement-factory.com> <3B7A7BDFFD286F49A19A5B35D26E08FB8F102D@srvmail2.aurenz.local> <87k46fwerj.fsf@gmail.com> Message-ID: <3B7A7BDFFD286F49A19A5B35D26E08FB8F103C@srvmail2.aurenz.local> Hello Dimitry, as Alex suggested I'm writing some bug reports at the moment covering the parts of the documentation that need to be updated. :-) Regards, Gerrit -----Urspr?ngliche Nachricht----- Von: Dmitry Kurochkin [mailto:dmitry.kurochkin at measurement-factory.com] Gesendet: Freitag, 2. Dezember 2011 09:37 An: Hohl, Gerrit; users at web-polygraph.org Betreff: Re: AW: AW: polygraph-pgl2acl doesn't recognize credentials Hi Gerrit. On Fri, 2 Dec 2011 09:08:29 +0100, "Hohl, Gerrit" wrote: > Hello Alex, > > > Sorry, some Polygraph features are not yet fully documented. Feel > > free to file a bug report for the ones you noticed -- it will help > > developers to keep track of what they need to fix: > > https://bugs.launchpad.net/polygraph > > okay, I will do that. :-) > Also I'm not sure in all cases if it is a problem of the documentation > or if it depends on other things (e.g. in the Ubuntu polygraph package > the names of the executables are different and polypxy doesn't exist). > It is outdated documentation. I maintain Debian (and Ubuntu) Web Polygraph package. The package does not do any renames or other weird stuff. You would get the same executable names if you compile by hand and do make install. In Web Polygraph v4.0.8 many executables were renamed and "polygraph-" prefix is now added during install. In particular, polyclt and polysrv are polygraph-client and polygraph-server now. Unfortunately, documentation is not yet fully updated. As for polypxy tool, it has not been available for a long time. Again, documentation is not up-to-date here. Regards, Dmitry > > Sure, that approach should work. Please also consider adding a > > DES(?) wrapping function to the pgl2ldif program instead. It would > > be a generally useful addition that allows anybody to easily produce > > htpasswd-style files from PGL workloads. > > I have to admit that I wrote a small Java program to solve that problem instead of modifying the C code of web polygraph. It uses the htpasswd executable (from the apache-util package) for creating the apache-style password files. > > > > Regards, > Gerrit > > -----Urspr?ngliche Nachricht----- > Von: Alex Rousskov [mailto:rousskov at measurement-factory.com] > Gesendet: Donnerstag, 1. Dezember 2011 17:10 > An: Hohl, Gerrit > Cc: users at web-polygraph.org > Betreff: Re: AW: polygraph-pgl2acl doesn't recognize credentials > > On 12/01/2011 01:24 AM, Hohl, Gerrit wrote: > > > I'm a little bit confused as there is no ACL property in the robot type: > > http://www.web-polygraph.org/docs/reference/pgl/types.html#type:docs > > /r eference/pgl/types/Robot By the way: There is also no credentials > > property or even a credentials(int, String) call in the documentation. > > ;-) > > Sorry, some Polygraph features are not yet fully documented. Feel free to file a bug report for the ones you noticed -- it will help developers to keep track of what they need to fix: > https://bugs.launchpad.net/polygraph > > > > Hm, I guess in this case it is easier for me to write a little > > program that creates a file which I can include in the proxy. That > > program will create another pgl file which I can include in the test > > case and use the array in the robot. > > Sure, that approach should work. Please also consider adding a DES(?) wrapping function to the pgl2ldif program instead. It would be a generally useful addition that allows anybody to easily produce htpasswd-style files from PGL workloads. > > > Cheers, > > Alex. > _______________________________________________ > Users mailing list > Users at web-polygraph.org > http://www.web-polygraph.org/mailman/listinfo/users From g.hohl at aurenz.de Mon Dec 5 12:11:22 2011 From: g.hohl at aurenz.de (Hohl, Gerrit) Date: Mon, 5 Dec 2011 13:11:22 +0100 Subject: polygraph-client tries to connect to server with odd IP address Message-ID: <3B7A7BDFFD286F49A19A5B35D26E08FB8F116A@srvmail2.aurenz.local> Hello everyone, I've build a test case and facing now the problem that the polygraph-client tries to connect to server which don't exist. That is somehow odd as the polygraph-server and the polygraph-client using the same PGL file. If they use the same file and use the same PGL parser, shouldn't they work with the same numbers / IP addresses for servers and clients? I'm a little bit confused. At the end of this message is the content of the PGL file. cred.pg contains some user credentials, but I don't use them at the moment. DNS names are currently not used (or at least should not be used). I only uses aliases in this version, no real machines (except the one where all the aliases are configured). polygraph-server starts 26 agents, from 10.101.128.1 to 10.101.128.26. polygraph-client starts 6 agents, from 10.101.0.1 to 10.101.0.3, 2 on each alias. After the start of the test case polygraph-client prints, e.g., that it can't establish a connection from 10.101.0.1:38686 to 10.101.148.15:80. How does it come up with that server IP address? It is not even in the addr_space of the server. :-? Make I should mention that I configure the aliases with the polygraph-aka command: polygraph-aka --if lo --aliases 10.101.0.1-125/22 --aliases 10.101.128-130.1-250/22 This creates 125 aliases for the servers and 750 aliases for the clients. // this is just one of the simplest workloads that can produce hits // never use this workload for benchmarking #include "cred.pg" // SimpleContent defines properties of content that the server generates; // if you get no hits, set SimpleContent.obj_life_cycle to cntStatic, which // is defined in workloads/include/contents.pg Content simpleContent = { size = exp(13KB); // response sizes distributed exponentially cachable = 80%; // 20% of content is uncachable }; Bench theBench = { peak_req_rate = 10000/sec; client_side = { max_host_load = 500/sec; max_agent_load = 100/sec; addr_space = [ 'lo::10.101.0-123.1-250/22' ]; }; server_side = { max_host_load = client_side.max_host_load; max_agent_load = undef(); addr_space = [ 'lo::10.101.128-129.1-250:80/22' ]; }; }; PolyMix4As asPolyMix4 = { agents_per_addr = 2; }; DnsResolver resolver = { servers = [ '127.0.0.1:53' ]; timeout = 5sec; }; Server server = { kind = "simple-srv"; contents = [ simpleContent ]; direct_access = contents; addresses = serverAddrs(asPolyMix4, theBench); }; AddrMap addrMap = { zone = "bench.tst"; addresses = server.addresses; names = ipsToNames(addresses, zone); }; // a primitive robot Robot robot = { kind = "simple-rbt"; pop_model = { pop_distr = popUnif(); }; recurrence = 55% / simpleContent.cachable; // adjusted to get 55% DHR // dns_resolver = resolver; // origins = addrMap.names; origins = server.addresses; addresses = robotAddrs(asPolyMix4, theBench); // credentials = cred; }; Phase inc = { name = "inc"; goal.duration = 5min; populus_factor_beg = 0%; populus_factor_end = 100%; load_factor_beg = 0%; load_factor_end = 100%; }; Phase top = { name = "top"; goal.duration = 10min; }; Phase dec = { name = "dec"; goal.duration = 5min; populus_factor_beg = 100%; populus_factor_end = 0%; load_factor_beg = 100%; load_factor_end = 0%; }; schedule(inc, top, dec); // commit to using these servers and robots use(server, robot); use(addrMap); use(theBench); From g.hohl at aurenz.de Mon Dec 5 12:24:55 2011 From: g.hohl at aurenz.de (Hohl, Gerrit) Date: Mon, 5 Dec 2011 13:24:55 +0100 Subject: AW: polygraph-client tries to connect to server with odd IP address In-Reply-To: <3B7A7BDFFD286F49A19A5B35D26E08FB8F116A@srvmail2.aurenz.local> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8F116A@srvmail2.aurenz.local> Message-ID: <3B7A7BDFFD286F49A19A5B35D26E08FB8F1173@srvmail2.aurenz.local> I changed the addr_space of the server side so it is now covering also these odd IP addresses. But it still starts only 26 server agents and the error messages still appear. :-( Bench theBench = { peak_req_rate = 10000/sec; client_side = { max_host_load = 500/sec; max_agent_load = 100/sec; addr_space = [ 'lo::10.101.0-123.1-250/22' ]; }; server_side = { max_host_load = client_side.max_host_load; max_agent_load = undef(); addr_space = [ 'lo::10.101.128-251.1-250:80/22' ]; }; }; -----Urspr?ngliche Nachricht----- Von: users-bounces at web-polygraph.org [mailto:users-bounces at web-polygraph.org] Im Auftrag von Hohl, Gerrit Gesendet: Montag, 5. Dezember 2011 13:11 An: users at web-polygraph.org Betreff: polygraph-client tries to connect to server with odd IP address Hello everyone, I've build a test case and facing now the problem that the polygraph-client tries to connect to server which don't exist. That is somehow odd as the polygraph-server and the polygraph-client using the same PGL file. If they use the same file and use the same PGL parser, shouldn't they work with the same numbers / IP addresses for servers and clients? I'm a little bit confused. At the end of this message is the content of the PGL file. cred.pg contains some user credentials, but I don't use them at the moment. DNS names are currently not used (or at least should not be used). I only uses aliases in this version, no real machines (except the one where all the aliases are configured). polygraph-server starts 26 agents, from 10.101.128.1 to 10.101.128.26. polygraph-client starts 6 agents, from 10.101.0.1 to 10.101.0.3, 2 on each alias. After the start of the test case polygraph-client prints, e.g., that it can't establish a connection from 10.101.0.1:38686 to 10.101.148.15:80. How does it come up with that server IP address? It is not even in the addr_space of the server. :-? Make I should mention that I configure the aliases with the polygraph-aka command: polygraph-aka --if lo --aliases 10.101.0.1-125/22 --aliases 10.101.128-130.1-250/22 This creates 125 aliases for the servers and 750 aliases for the clients. // this is just one of the simplest workloads that can produce hits // never use this workload for benchmarking #include "cred.pg" // SimpleContent defines properties of content that the server generates; // if you get no hits, set SimpleContent.obj_life_cycle to cntStatic, which // is defined in workloads/include/contents.pg Content simpleContent = { size = exp(13KB); // response sizes distributed exponentially cachable = 80%; // 20% of content is uncachable }; Bench theBench = { peak_req_rate = 10000/sec; client_side = { max_host_load = 500/sec; max_agent_load = 100/sec; addr_space = [ 'lo::10.101.0-123.1-250/22' ]; }; server_side = { max_host_load = client_side.max_host_load; max_agent_load = undef(); addr_space = [ 'lo::10.101.128-129.1-250:80/22' ]; }; }; PolyMix4As asPolyMix4 = { agents_per_addr = 2; }; DnsResolver resolver = { servers = [ '127.0.0.1:53' ]; timeout = 5sec; }; Server server = { kind = "simple-srv"; contents = [ simpleContent ]; direct_access = contents; addresses = serverAddrs(asPolyMix4, theBench); }; AddrMap addrMap = { zone = "bench.tst"; addresses = server.addresses; names = ipsToNames(addresses, zone); }; // a primitive robot Robot robot = { kind = "simple-rbt"; pop_model = { pop_distr = popUnif(); }; recurrence = 55% / simpleContent.cachable; // adjusted to get 55% DHR // dns_resolver = resolver; // origins = addrMap.names; origins = server.addresses; addresses = robotAddrs(asPolyMix4, theBench); // credentials = cred; }; Phase inc = { name = "inc"; goal.duration = 5min; populus_factor_beg = 0%; populus_factor_end = 100%; load_factor_beg = 0%; load_factor_end = 100%; }; Phase top = { name = "top"; goal.duration = 10min; }; Phase dec = { name = "dec"; goal.duration = 5min; populus_factor_beg = 100%; populus_factor_end = 0%; load_factor_beg = 100%; load_factor_end = 0%; }; schedule(inc, top, dec); // commit to using these servers and robots use(server, robot); use(addrMap); use(theBench); _______________________________________________ Users mailing list Users at web-polygraph.org http://www.web-polygraph.org/mailman/listinfo/users From dmitry.kurochkin at measurement-factory.com Mon Dec 5 13:12:35 2011 From: dmitry.kurochkin at measurement-factory.com (Dmitry Kurochkin) Date: Mon, 05 Dec 2011 17:12:35 +0400 Subject: polygraph-client tries to connect to server with odd IP address In-Reply-To: <3B7A7BDFFD286F49A19A5B35D26E08FB8F116A@srvmail2.aurenz.local> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8F116A@srvmail2.aurenz.local> Message-ID: <87liqrtb5o.fsf@gmail.com> Hi Gerrit. On Mon, 5 Dec 2011 13:11:22 +0100, "Hohl, Gerrit" wrote: > Hello everyone, > > I've build a test case and facing now the problem that the > polygraph-client tries to connect to server which don't exist. That is > somehow odd as the polygraph-server and the polygraph-client using the > same PGL file. If they use the same file and use the same PGL parser, > shouldn't they work with the same numbers / IP addresses for servers and > clients? I'm a little bit confused. > Yes, it should. > At the end of this message is the content of the PGL file. > cred.pg contains some user credentials, but I don't use them at the > moment. > DNS names are currently not used (or at least should not be used). > I only uses aliases in this version, no real machines (except the one > where all the aliases are configured). > polygraph-server starts 26 agents, from 10.101.128.1 to 10.101.128.26. > polygraph-client starts 6 agents, from 10.101.0.1 to 10.101.0.3, 2 on > each alias. > After the start of the test case polygraph-client prints, e.g., that it > can't establish a connection from 10.101.0.1:38686 to 10.101.148.15:80. > How does it come up with that server IP address? It is not even in the > addr_space of the server. :-? There are no hard-coded addresses in Polygraph, but some standard workloads use 10.101.128-251.1-250 server address space by default. Please double check that you are running the correct workload. Note that the workload you sent does not work as is because the server address space is too small: serverAddrs() call failed: too small address space; 520 agents cannot fit into 500 space slots Regards, Dmitry > Make I should mention that I configure the aliases with the > polygraph-aka command: > polygraph-aka --if lo --aliases 10.101.0.1-125/22 --aliases > 10.101.128-130.1-250/22 > This creates 125 aliases for the servers and 750 aliases for the > clients. > > > // this is just one of the simplest workloads that can produce hits > // never use this workload for benchmarking > #include "cred.pg" > > // SimpleContent defines properties of content that the server > generates; > // if you get no hits, set SimpleContent.obj_life_cycle to cntStatic, > which > // is defined in workloads/include/contents.pg > Content simpleContent = { > size = exp(13KB); // response sizes distributed exponentially > cachable = 80%; // 20% of content is uncachable > }; > > Bench theBench = { > peak_req_rate = 10000/sec; > > client_side = { > max_host_load = 500/sec; > max_agent_load = 100/sec; > addr_space = [ 'lo::10.101.0-123.1-250/22' ]; > > }; > server_side = { > max_host_load = client_side.max_host_load; > max_agent_load = undef(); > addr_space = [ 'lo::10.101.128-129.1-250:80/22' ]; > }; > }; > > PolyMix4As asPolyMix4 = { > agents_per_addr = 2; > }; > > DnsResolver resolver = { > servers = [ '127.0.0.1:53' ]; > timeout = 5sec; > }; > > Server server = { > kind = "simple-srv"; > contents = [ simpleContent ]; > direct_access = contents; > > addresses = serverAddrs(asPolyMix4, theBench); > }; > > AddrMap addrMap = { > zone = "bench.tst"; > addresses = server.addresses; > names = ipsToNames(addresses, zone); > }; > > // a primitive robot > Robot robot = { > kind = "simple-rbt"; > pop_model = { pop_distr = popUnif(); }; > recurrence = 55% / simpleContent.cachable; // adjusted to get > 55% DHR > > // dns_resolver = resolver; > // origins = addrMap.names; > origins = server.addresses; > addresses = robotAddrs(asPolyMix4, theBench); > > // credentials = cred; > }; > > Phase inc = { > name = "inc"; > goal.duration = 5min; > populus_factor_beg = 0%; > populus_factor_end = 100%; > load_factor_beg = 0%; > load_factor_end = 100%; > }; > > Phase top = { > name = "top"; > goal.duration = 10min; > }; > > Phase dec = { > name = "dec"; > goal.duration = 5min; > populus_factor_beg = 100%; > populus_factor_end = 0%; > load_factor_beg = 100%; > load_factor_end = 0%; > }; > > schedule(inc, top, dec); > > // commit to using these servers and robots > use(server, robot); > use(addrMap); > use(theBench); > _______________________________________________ > Users mailing list > Users at web-polygraph.org > http://www.web-polygraph.org/mailman/listinfo/users From g.hohl at aurenz.de Mon Dec 5 15:51:36 2011 From: g.hohl at aurenz.de (Hohl, Gerrit) Date: Mon, 5 Dec 2011 16:51:36 +0100 Subject: AW: polygraph-client tries to connect to server with odd IP address In-Reply-To: <87liqrtb5o.fsf@gmail.com> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8F116A@srvmail2.aurenz.local> <87liqrtb5o.fsf@gmail.com> Message-ID: <3B7A7BDFFD286F49A19A5B35D26E08FB8F11AF@srvmail2.aurenz.local> Hello Dimitry, I guess I found the problem: The ratio between peak_req_rate, max_host_load, max_agent_load and agents_per_addr is the root of all. If the relationship of this 4 properties is not in balance polygraph-client somehow assumes a wrong number of servers. Currently if've the following address space for the client side: addr_space = [ 'lo::10.101.0-123.1-250/22' ]; and the following for the server side: addr_space = [ 'lo::10.101.128-251.1-250:80/22' ]; And it is working perfectly now. :-) I only use this configuration so I can "play" with web polygraph on a Ubuntu 10.04.3 server installation and prepare a load test / stress test. But the real one will be run on real hardware and also on a few computers instead of only one. So I will change the network configuration later. Regards, Gerrit -----Urspr?ngliche Nachricht----- Von: Dmitry Kurochkin [mailto:dmitry.kurochkin at measurement-factory.com] Gesendet: Montag, 5. Dezember 2011 14:13 An: Hohl, Gerrit; users at web-polygraph.org Betreff: Re: polygraph-client tries to connect to server with odd IP address Hi Gerrit. On Mon, 5 Dec 2011 13:11:22 +0100, "Hohl, Gerrit" wrote: > Hello everyone, > > I've build a test case and facing now the problem that the > polygraph-client tries to connect to server which don't exist. That is > somehow odd as the polygraph-server and the polygraph-client using the > same PGL file. If they use the same file and use the same PGL parser, > shouldn't they work with the same numbers / IP addresses for servers > and clients? I'm a little bit confused. > Yes, it should. > At the end of this message is the content of the PGL file. > cred.pg contains some user credentials, but I don't use them at the > moment. > DNS names are currently not used (or at least should not be used). > I only uses aliases in this version, no real machines (except the one > where all the aliases are configured). > polygraph-server starts 26 agents, from 10.101.128.1 to 10.101.128.26. > polygraph-client starts 6 agents, from 10.101.0.1 to 10.101.0.3, 2 on > each alias. > After the start of the test case polygraph-client prints, e.g., that > it can't establish a connection from 10.101.0.1:38686 to 10.101.148.15:80. > How does it come up with that server IP address? It is not even in the > addr_space of the server. :-? There are no hard-coded addresses in Polygraph, but some standard workloads use 10.101.128-251.1-250 server address space by default. Please double check that you are running the correct workload. Note that the workload you sent does not work as is because the server address space is too small: serverAddrs() call failed: too small address space; 520 agents cannot fit into 500 space slots Regards, Dmitry > Make I should mention that I configure the aliases with the > polygraph-aka command: > polygraph-aka --if lo --aliases 10.101.0.1-125/22 --aliases > 10.101.128-130.1-250/22 > This creates 125 aliases for the servers and 750 aliases for the > clients. > > > // this is just one of the simplest workloads that can produce hits // > never use this workload for benchmarking #include "cred.pg" > > // SimpleContent defines properties of content that the server > generates; // if you get no hits, set SimpleContent.obj_life_cycle to > cntStatic, which // is defined in workloads/include/contents.pg > Content simpleContent = { > size = exp(13KB); // response sizes distributed exponentially > cachable = 80%; // 20% of content is uncachable > }; > > Bench theBench = { > peak_req_rate = 10000/sec; > > client_side = { > max_host_load = 500/sec; > max_agent_load = 100/sec; > addr_space = [ 'lo::10.101.0-123.1-250/22' ]; > > }; > server_side = { > max_host_load = client_side.max_host_load; > max_agent_load = undef(); > addr_space = [ 'lo::10.101.128-129.1-250:80/22' ]; > }; > }; > > PolyMix4As asPolyMix4 = { > agents_per_addr = 2; > }; > > DnsResolver resolver = { > servers = [ '127.0.0.1:53' ]; > timeout = 5sec; > }; > > Server server = { > kind = "simple-srv"; > contents = [ simpleContent ]; > direct_access = contents; > > addresses = serverAddrs(asPolyMix4, theBench); }; > > AddrMap addrMap = { > zone = "bench.tst"; > addresses = server.addresses; > names = ipsToNames(addresses, zone); > }; > > // a primitive robot > Robot robot = { > kind = "simple-rbt"; > pop_model = { pop_distr = popUnif(); }; > recurrence = 55% / simpleContent.cachable; // adjusted to get 55% DHR > > // dns_resolver = resolver; > // origins = addrMap.names; > origins = server.addresses; > addresses = robotAddrs(asPolyMix4, theBench); > > // credentials = cred; > }; > > Phase inc = { > name = "inc"; > goal.duration = 5min; > populus_factor_beg = 0%; > populus_factor_end = 100%; > load_factor_beg = 0%; > load_factor_end = 100%; > }; > > Phase top = { > name = "top"; > goal.duration = 10min; > }; > > Phase dec = { > name = "dec"; > goal.duration = 5min; > populus_factor_beg = 100%; > populus_factor_end = 0%; > load_factor_beg = 100%; > load_factor_end = 0%; > }; > > schedule(inc, top, dec); > > // commit to using these servers and robots use(server, robot); > use(addrMap); use(theBench); > _______________________________________________ > Users mailing list > Users at web-polygraph.org > http://www.web-polygraph.org/mailman/listinfo/users From dmitry.kurochkin at measurement-factory.com Mon Dec 5 17:03:28 2011 From: dmitry.kurochkin at measurement-factory.com (Dmitry Kurochkin) Date: Mon, 05 Dec 2011 21:03:28 +0400 Subject: AW: polygraph-client tries to connect to server with odd IP address In-Reply-To: <3B7A7BDFFD286F49A19A5B35D26E08FB8F11AF@srvmail2.aurenz.local> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8F116A@srvmail2.aurenz.local> <87liqrtb5o.fsf@gmail.com> <3B7A7BDFFD286F49A19A5B35D26E08FB8F11AF@srvmail2.aurenz.local> Message-ID: <87fwgzt0gv.fsf@gmail.com> On Mon, 5 Dec 2011 16:51:36 +0100, "Hohl, Gerrit" wrote: > Hello Dimitry, > > I guess I found the problem: > The ratio between peak_req_rate, max_host_load, max_agent_load and > agents_per_addr is the root of all. If the relationship of this 4 > properties is not in balance polygraph-client somehow assumes a wrong > number of servers. > Yes, the number of agents is calculated by Polygraph based on rate and load values. But if the configuration is not valid (e.g. required number of agents exceeds the address space), Polygraph should exit with an error. Otherwise it may be a bug. I would like to reproduce the error you get. The workload in your first email does not show the problem with Polygraph v4.3.2: Polygraph exits with the expected "too small address space" error. What version of Web Polygraph are you using? Regards, Dmitry > Currently if've the following address space for the client side: > > addr_space = [ 'lo::10.101.0-123.1-250/22' ]; > > and the following for the server side: > > addr_space = [ 'lo::10.101.128-251.1-250:80/22' ]; > > And it is working perfectly now. :-) > > I only use this configuration so I can "play" with web polygraph on a Ubuntu 10.04.3 server installation and prepare a load test / stress test. But the real one will be run on real hardware and also on a few computers instead of only one. So I will change the network configuration later. > > Regards, > Gerrit > > -----Urspr?ngliche Nachricht----- > Von: Dmitry Kurochkin [mailto:dmitry.kurochkin at measurement-factory.com] > Gesendet: Montag, 5. Dezember 2011 14:13 > An: Hohl, Gerrit; users at web-polygraph.org > Betreff: Re: polygraph-client tries to connect to server with odd IP address > > Hi Gerrit. > > On Mon, 5 Dec 2011 13:11:22 +0100, "Hohl, Gerrit" wrote: > > Hello everyone, > > > > I've build a test case and facing now the problem that the > > polygraph-client tries to connect to server which don't exist. That is > > somehow odd as the polygraph-server and the polygraph-client using the > > same PGL file. If they use the same file and use the same PGL parser, > > shouldn't they work with the same numbers / IP addresses for servers > > and clients? I'm a little bit confused. > > > > Yes, it should. > > > At the end of this message is the content of the PGL file. > > cred.pg contains some user credentials, but I don't use them at the > > moment. > > DNS names are currently not used (or at least should not be used). > > I only uses aliases in this version, no real machines (except the one > > where all the aliases are configured). > > polygraph-server starts 26 agents, from 10.101.128.1 to 10.101.128.26. > > polygraph-client starts 6 agents, from 10.101.0.1 to 10.101.0.3, 2 on > > each alias. > > After the start of the test case polygraph-client prints, e.g., that > > it can't establish a connection from 10.101.0.1:38686 to 10.101.148.15:80. > > How does it come up with that server IP address? It is not even in the > > addr_space of the server. :-? > > There are no hard-coded addresses in Polygraph, but some standard workloads use 10.101.128-251.1-250 server address space by default. > Please double check that you are running the correct workload. > > Note that the workload you sent does not work as is because the server address space is too small: > > serverAddrs() call failed: too small address space; 520 agents cannot fit into 500 space slots > > Regards, > Dmitry > > > Make I should mention that I configure the aliases with the > > polygraph-aka command: > > polygraph-aka --if lo --aliases 10.101.0.1-125/22 --aliases > > 10.101.128-130.1-250/22 > > This creates 125 aliases for the servers and 750 aliases for the > > clients. > > > > > > // this is just one of the simplest workloads that can produce hits // > > never use this workload for benchmarking #include "cred.pg" > > > > // SimpleContent defines properties of content that the server > > generates; // if you get no hits, set SimpleContent.obj_life_cycle to > > cntStatic, which // is defined in workloads/include/contents.pg > > Content simpleContent = { > > size = exp(13KB); // response sizes distributed exponentially > > cachable = 80%; // 20% of content is uncachable > > }; > > > > Bench theBench = { > > peak_req_rate = 10000/sec; > > > > client_side = { > > max_host_load = 500/sec; > > max_agent_load = 100/sec; > > addr_space = [ 'lo::10.101.0-123.1-250/22' ]; > > > > }; > > server_side = { > > max_host_load = client_side.max_host_load; > > max_agent_load = undef(); > > addr_space = [ 'lo::10.101.128-129.1-250:80/22' ]; > > }; > > }; > > > > PolyMix4As asPolyMix4 = { > > agents_per_addr = 2; > > }; > > > > DnsResolver resolver = { > > servers = [ '127.0.0.1:53' ]; > > timeout = 5sec; > > }; > > > > Server server = { > > kind = "simple-srv"; > > contents = [ simpleContent ]; > > direct_access = contents; > > > > addresses = serverAddrs(asPolyMix4, theBench); }; > > > > AddrMap addrMap = { > > zone = "bench.tst"; > > addresses = server.addresses; > > names = ipsToNames(addresses, zone); > > }; > > > > // a primitive robot > > Robot robot = { > > kind = "simple-rbt"; > > pop_model = { pop_distr = popUnif(); }; > > recurrence = 55% / simpleContent.cachable; // adjusted to get 55% DHR > > > > // dns_resolver = resolver; > > // origins = addrMap.names; > > origins = server.addresses; > > addresses = robotAddrs(asPolyMix4, theBench); > > > > // credentials = cred; > > }; > > > > Phase inc = { > > name = "inc"; > > goal.duration = 5min; > > populus_factor_beg = 0%; > > populus_factor_end = 100%; > > load_factor_beg = 0%; > > load_factor_end = 100%; > > }; > > > > Phase top = { > > name = "top"; > > goal.duration = 10min; > > }; > > > > Phase dec = { > > name = "dec"; > > goal.duration = 5min; > > populus_factor_beg = 100%; > > populus_factor_end = 0%; > > load_factor_beg = 100%; > > load_factor_end = 0%; > > }; > > > > schedule(inc, top, dec); > > > > // commit to using these servers and robots use(server, robot); > > use(addrMap); use(theBench); > > _______________________________________________ > > Users mailing list > > Users at web-polygraph.org > > http://www.web-polygraph.org/mailman/listinfo/users > _______________________________________________ > Users mailing list > Users at web-polygraph.org > http://www.web-polygraph.org/mailman/listinfo/users From g.hohl at aurenz.de Tue Dec 6 08:02:38 2011 From: g.hohl at aurenz.de (Hohl, Gerrit) Date: Tue, 6 Dec 2011 09:02:38 +0100 Subject: AW: AW: polygraph-client tries to connect to server with odd IP address In-Reply-To: <87fwgzt0gv.fsf@gmail.com> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8F116A@srvmail2.aurenz.local> <87liqrtb5o.fsf@gmail.com> <3B7A7BDFFD286F49A19A5B35D26E08FB8F11AF@srvmail2.aurenz.local> <87fwgzt0gv.fsf@gmail.com> Message-ID: <3B7A7BDFFD286F49A19A5B35D26E08FB8F11D9@srvmail2.aurenz.local> Good morning Dmitry, here is the current test case I'm using. With these values it works perfectly. But if change the peak_req_rate to 25000/sec, the polygraph-server will start 567 servers while the polygraph-client starts 250 robots, expecting 1126 servers. I guess it is not valid to have a bigger peak_req_rate than the max_host_load. But instead of using a false number of servers (shouldn't it be the same for client as well as for the server?) maybe it should quit by showing an error that the parameters are not valid? /* * The focus of this test is response time and bandwidth under high load. * Server and robot don't have a think time or anything like that. They just * put as much stress as possible on the proxy. */ #include "contents.pg" #include "cred.pg" Bench theBench = { // The peak request rate is 10.000 requests per second peak_req_rate = 10000/sec; client_side = { /* * The servers should be able to handle 20.000 requests per * second. */ max_host_load = 20000/sec; /* * Every robot should create 4 requests per seconds. */ max_agent_load = 4/sec; addr_space = [ 'lo::10.101.0-123.1-250/22' ]; }; server_side = { max_host_load = client_side.max_host_load; max_agent_load = undef(); addr_space = [ 'lo::10.101.128-251.1-250:80/22' ]; }; }; PolyMix4As asPolyMix4 = { // We will have 2 robots on each IP address / alias. agents_per_addr = 2; }; DnsResolver resolver = { servers = [ '127.0.0.1:53' ]; timeout = 5sec; }; Server server = { kind = "simple-srv"; contents = [ cntImage: 65%, cntHTML: 15%, cntDownload: 0.5%, cntOther ]; direct_access = [ cntHTML, cntDownload, cntOther ]; addresses = serverAddrs(asPolyMix4, theBench); }; AddrMap addrMap = { zone = "bench.tst"; addresses = server.addresses; names = ipsToNames(addresses, zone); }; // a primitive robot Robot robot = { kind = "simple-rbt"; pop_model = { pop_distr = popUnif(); }; req_types = [ "Basic", "Ims200": 5%, "Ims304": 10%, "Reload": 5% ]; req_methods = [ "GET", "POST": 1.5%, "HEAD": 0.1% ]; dns_resolver = resolver; origins = addrMap.names; // origins = server.addresses; addresses = robotAddrs(asPolyMix4, theBench); post_contents = [ cntSimpleRequest ]; credentials = cred; }; Phase inc = { name = "inc"; goal.duration = 5min; populus_factor_beg = 0%; populus_factor_end = 100%; load_factor_beg = 0%; load_factor_end = 100%; }; Phase top = { name = "top"; goal.duration = 10min; }; Phase dec = { name = "dec"; goal.duration = 5min; populus_factor_beg = 100%; populus_factor_end = 0%; load_factor_beg = 100%; load_factor_end = 0%; }; schedule(inc, top, dec); // commit to using these servers and robots use(server, robot); use(addrMap); use(theBench); -----Urspr?ngliche Nachricht----- Von: Dmitry Kurochkin [mailto:dmitry.kurochkin at measurement-factory.com] Gesendet: Montag, 5. Dezember 2011 18:03 An: Hohl, Gerrit; users at web-polygraph.org Betreff: Re: AW: polygraph-client tries to connect to server with odd IP address On Mon, 5 Dec 2011 16:51:36 +0100, "Hohl, Gerrit" wrote: > Hello Dimitry, > > I guess I found the problem: > The ratio between peak_req_rate, max_host_load, max_agent_load and > agents_per_addr is the root of all. If the relationship of this 4 > properties is not in balance polygraph-client somehow assumes a wrong > number of servers. > Yes, the number of agents is calculated by Polygraph based on rate and load values. But if the configuration is not valid (e.g. required number of agents exceeds the address space), Polygraph should exit with an error. Otherwise it may be a bug. I would like to reproduce the error you get. The workload in your first email does not show the problem with Polygraph v4.3.2: Polygraph exits with the expected "too small address space" error. What version of Web Polygraph are you using? Regards, Dmitry > Currently if've the following address space for the client side: > > addr_space = [ 'lo::10.101.0-123.1-250/22' ]; > > and the following for the server side: > > addr_space = [ 'lo::10.101.128-251.1-250:80/22' ]; > > And it is working perfectly now. :-) > > I only use this configuration so I can "play" with web polygraph on a Ubuntu 10.04.3 server installation and prepare a load test / stress test. But the real one will be run on real hardware and also on a few computers instead of only one. So I will change the network configuration later. > > Regards, > Gerrit > > -----Urspr?ngliche Nachricht----- > Von: Dmitry Kurochkin > [mailto:dmitry.kurochkin at measurement-factory.com] > Gesendet: Montag, 5. Dezember 2011 14:13 > An: Hohl, Gerrit; users at web-polygraph.org > Betreff: Re: polygraph-client tries to connect to server with odd IP > address > > Hi Gerrit. > > On Mon, 5 Dec 2011 13:11:22 +0100, "Hohl, Gerrit" wrote: > > Hello everyone, > > > > I've build a test case and facing now the problem that the > > polygraph-client tries to connect to server which don't exist. That > > is somehow odd as the polygraph-server and the polygraph-client > > using the same PGL file. If they use the same file and use the same > > PGL parser, shouldn't they work with the same numbers / IP addresses > > for servers and clients? I'm a little bit confused. > > > > Yes, it should. > > > At the end of this message is the content of the PGL file. > > cred.pg contains some user credentials, but I don't use them at the > > moment. > > DNS names are currently not used (or at least should not be used). > > I only uses aliases in this version, no real machines (except the > > one where all the aliases are configured). > > polygraph-server starts 26 agents, from 10.101.128.1 to 10.101.128.26. > > polygraph-client starts 6 agents, from 10.101.0.1 to 10.101.0.3, 2 > > on each alias. > > After the start of the test case polygraph-client prints, e.g., that > > it can't establish a connection from 10.101.0.1:38686 to 10.101.148.15:80. > > How does it come up with that server IP address? It is not even in > > the addr_space of the server. :-? > > There are no hard-coded addresses in Polygraph, but some standard workloads use 10.101.128-251.1-250 server address space by default. > Please double check that you are running the correct workload. > > Note that the workload you sent does not work as is because the server address space is too small: > > serverAddrs() call failed: too small address space; 520 agents > cannot fit into 500 space slots > > Regards, > Dmitry > > > Make I should mention that I configure the aliases with the > > polygraph-aka command: > > polygraph-aka --if lo --aliases 10.101.0.1-125/22 --aliases > > 10.101.128-130.1-250/22 > > This creates 125 aliases for the servers and 750 aliases for the > > clients. > > > > > > // this is just one of the simplest workloads that can produce hits > > // never use this workload for benchmarking #include "cred.pg" > > > > // SimpleContent defines properties of content that the server > > generates; // if you get no hits, set SimpleContent.obj_life_cycle > > to cntStatic, which // is defined in workloads/include/contents.pg > > Content simpleContent = { > > size = exp(13KB); // response sizes distributed exponentially > > cachable = 80%; // 20% of content is uncachable > > }; > > > > Bench theBench = { > > peak_req_rate = 10000/sec; > > > > client_side = { > > max_host_load = 500/sec; > > max_agent_load = 100/sec; > > addr_space = [ 'lo::10.101.0-123.1-250/22' ]; > > > > }; > > server_side = { > > max_host_load = client_side.max_host_load; > > max_agent_load = undef(); > > addr_space = [ 'lo::10.101.128-129.1-250:80/22' ]; > > }; > > }; > > > > PolyMix4As asPolyMix4 = { > > agents_per_addr = 2; > > }; > > > > DnsResolver resolver = { > > servers = [ '127.0.0.1:53' ]; > > timeout = 5sec; > > }; > > > > Server server = { > > kind = "simple-srv"; > > contents = [ simpleContent ]; > > direct_access = contents; > > > > addresses = serverAddrs(asPolyMix4, theBench); }; > > > > AddrMap addrMap = { > > zone = "bench.tst"; > > addresses = server.addresses; > > names = ipsToNames(addresses, zone); }; > > > > // a primitive robot > > Robot robot = { > > kind = "simple-rbt"; > > pop_model = { pop_distr = popUnif(); }; > > recurrence = 55% / simpleContent.cachable; // adjusted to get 55% > > DHR > > > > // dns_resolver = resolver; > > // origins = addrMap.names; > > origins = server.addresses; > > addresses = robotAddrs(asPolyMix4, theBench); > > > > // credentials = cred; > > }; > > > > Phase inc = { > > name = "inc"; > > goal.duration = 5min; > > populus_factor_beg = 0%; > > populus_factor_end = 100%; > > load_factor_beg = 0%; > > load_factor_end = 100%; > > }; > > > > Phase top = { > > name = "top"; > > goal.duration = 10min; > > }; > > > > Phase dec = { > > name = "dec"; > > goal.duration = 5min; > > populus_factor_beg = 100%; > > populus_factor_end = 0%; > > load_factor_beg = 100%; > > load_factor_end = 0%; > > }; > > > > schedule(inc, top, dec); > > > > // commit to using these servers and robots use(server, robot); > > use(addrMap); use(theBench); > > _______________________________________________ > > Users mailing list > > Users at web-polygraph.org > > http://www.web-polygraph.org/mailman/listinfo/users > _______________________________________________ > Users mailing list > Users at web-polygraph.org > http://www.web-polygraph.org/mailman/listinfo/users From g.hohl at aurenz.de Tue Dec 6 15:41:36 2011 From: g.hohl at aurenz.de (Hohl, Gerrit) Date: Tue, 6 Dec 2011 16:41:36 +0100 Subject: Changing the IP address range has an influence on the number of robots? Message-ID: <3B7A7BDFFD286F49A19A5B35D26E08FB8F1280@srvmail2.aurenz.local> Hello everyone, now I have an interesting phenomenon: Changing the IP address range seems to have an influence on the number of robots. I cloned my test image 4 times - for 2 clients and 2 servers. I modifed the systems, so no one of them is in conflict with another system. First I wanted to try to configure the test for 1 client and 1 server. I configured the aliases and routes and everything worked fine (including ping and so on). Then I modified my test by changing the IP address range in addr_space and added the hosts property. I paid attentation to that mapping thing which was described in polymix-4 configuration. I uncommented everything related to DNS so I don't have to care for that. Then I started server and client (without proxy). The server showed be that 750 virtual servers have been started - which is okay and exactly the same as before. But when I started the client, it showed my that it started 2500 robots. Previously it started only 250 - 2 on each of the 125 aliases. I uncommented the hosts property, but that didn't changed anything. Why does changing the IP address range have an effect on the number of robots? And how? /* * The focus of this test is response time and bandwidth under high load. * Server and robot don't have a think time or anything like that. They just * put as much stress as possible on the proxy. */ #include "contents.pg" #include "cred.pg" Bench theBench = { // The peak request rate is 10.000 requests per second peak_req_rate = 10000/sec; client_side = { /* * The servers should be able to handle 20.000 requests per * second. */ max_host_load = 20000/sec; /* * Every robot should create 4 requests per seconds. */ max_agent_load = 4/sec; addr_space = [ 'lo::10.5.0-123.1-250/22' ]; // hosts = [ '172.16.5.1' ]; }; server_side = { max_host_load = client_side.max_host_load; max_agent_load = undef(); addr_space = [ 'lo::10.5.128-251.1-250:80/22' ]; // hosts = [ '172.16.5.128' ]; }; }; PolyMix4As asPolyMix4 = { // We will have 2 robots on each IP address / alias. agents_per_addr = 2; }; DnsResolver resolver = { servers = [ '127.0.0.1:53' ]; timeout = 5sec; }; Server server = { kind = "simple-srv"; contents = [ cntImage: 65%, cntHTML: 15%, cntDownload: 0.5%, cntOther ]; direct_access = [ cntHTML, cntDownload, cntOther ]; addresses = serverAddrs(asPolyMix4, theBench); }; AddrMap addrMap = { zone = "bench.tst"; addresses = server.addresses; names = ipsToNames(addresses, zone); }; // a primitive robot Robot robot = { kind = "simple-rbt"; pop_model = { pop_distr = popUnif(); }; req_types = [ "Basic", "Ims200": 5%, "Ims304": 10%, "Reload": 5% ]; req_methods = [ "GET", "POST": 1.5%, "HEAD": 0.1% ]; dns_resolver = resolver; origins = addrMap.names; // origins = server.addresses; addresses = robotAddrs(asPolyMix4, theBench); post_contents = [ cntSimpleRequest ]; credentials = cred; }; Phase inc = { name = "inc"; goal.duration = 5min; populus_factor_beg = 0%; populus_factor_end = 100%; load_factor_beg = 0%; load_factor_end = 100%; }; Phase top = { name = "top"; goal.duration = 10min; }; Phase dec = { name = "dec"; goal.duration = 5min; populus_factor_beg = 100%; populus_factor_end = 0%; load_factor_beg = 100%; load_factor_end = 0%; }; schedule(inc, top, dec); // commit to using these servers and robots use(server, robot); use(addrMap); use(theBench); From dmitry.kurochkin at measurement-factory.com Tue Dec 6 17:47:51 2011 From: dmitry.kurochkin at measurement-factory.com (Dmitry Kurochkin) Date: Tue, 06 Dec 2011 21:47:51 +0400 Subject: Changing the IP address range has an influence on the number of robots? In-Reply-To: <3B7A7BDFFD286F49A19A5B35D26E08FB8F1280@srvmail2.aurenz.local> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8F1280@srvmail2.aurenz.local> Message-ID: <87y5upsibc.fsf@gmail.com> Hi Gerrit. I think you have some misunderstanding on how Polygraph calculates configured Agent addresses and how these addresses are used. I will try to explain how the things work below. There are several independent steps during Polygraph client and server startup: * calculating client and server addresses (serverAddrs() and robotAddrs() PGL functions) * network alias creation * starting up configured Agents The first step is optional - nothing is calculated if serverAddrs() or robotAddrs() PGL functions are not used. These functions do not depend on system configuration (e.g. configured network aliases) and they work in the same way on both client and server. I.e. for a given PGL file, serverAddrs() and robotAddrs() would produce the same results on any system in both polygraph-client and polygraph-server. Note that these functions are independent from the next two steps, you could replace these function calls with their results (array of addresses) and Polygraph would work in exactly the same way. Results of these functions depend only on the Bench configuration and the address scheme. The network alias creation step is optional as well. Polygraph creates aliases only for addresses with interface name and network mask. E.g., Polygraph would create alias for 'lo::10.10.10.10/24' address but would skip '10.10.10.10'. (IPv6 addresses do not need a netmask, in the future we may remove netmask requirement for IPv4 as well.) Polygraph also requires configured hosts for the bench side to create aliases. If there are no hosts configured, Polygraph does not create any aliases. You may see messages like these on the console: 000.01| found 6252 Robot IP address(es) total (3126 unique address(es) with interface name and netmask). 000.01| fyi: no real host addresses for Robot side specified That means that there are total 3126 aliases needed but Polygraph does not create them because no hosts are configured. The hosts are needed for Polygraph to decide what aliases it needs to create on the system. All aliases are evenly distributed between the configured hosts. E.g. if Robot has configured addresses ['lo::10.10.10.1-10/24'] and client-side hosts are ['192.168.1.1-2'], then polygraph-client would create aliases 10.10.10.1-5 when started on 192.168.1.1 and aliases 10.10.10.6-10 when started on 192.168.1.2. Polygraph would not create any addresses when started on a system which does not have any interface address matching the configured hosts addresses (but that is not an error, Polygraph would continue as usual). If you set hosts to a single address, Polygraph would create all configured aliases on that host. (One thing to note is that Polygraph removes all existing aliases before creating new ones unless "--delete_old_addrs false" option is given.) The next step is to start configured Agents. It is independent from both of the previous steps, i.e. it does not matter if you are using robotAddrs()/serverAddrs() or how aliases are created. At this point Polygraph has a list of agents (robots on client and servers of agents) with configured addresses. These addresses may or may not be configured on the system. Polygraph can not start an agent if the system does not have the corresponding network interface or alias. So it starts only the agents which have addresses configured on the system, others are just ignored. This allows you to run multiple client and server processes on different systems. E.g., you may set server addresses to '192.168.1.1-2' and run two polygraph-server processes on 192.168.1.1 and 192.168.1.2. Polygraph does not know whether you started servers on all the systems so it can not warn you about it. But you will get errors from Robots trying to connect to Servers which are configured but not started. The same is true for Robots, except that you do not get any errors during the test. So when running a test, make sure you: * Create all needed aliases on the systems you want to use or allow Polygraph to create aliases for you. * Run Polygraph client (server) on each configured host. * Pay attention to error messages printed on the console. E.g.: found 6252 Robot IP address(es) total (3126 unique address(es) with interface name and netmask). fyi: no real host addresses for Robot side specified created 250 agents total shows that the host has network interfaces for only 250 agents, while there are 6252 Robots configured. You either need to configure more aliases on the host, or start client processes on other systems which have these aliases. Going back to your original question: > Why does changing the IP address range have an effect on the number of > robots? And how? Changing Bench address space has no effect on the number of Robots, i.e. the number of results returned by robotAddrs(). The number of Robots changed because now Polygraph creates all aliases for you. Before you did not create all the needed aliases and Polygraph did not start all the configured Robots (so the test load was much less than what you expected). HTH, Dmitry From dmitry.kurochkin at measurement-factory.com Tue Dec 6 17:56:51 2011 From: dmitry.kurochkin at measurement-factory.com (Dmitry Kurochkin) Date: Tue, 06 Dec 2011 21:56:51 +0400 Subject: AW: AW: polygraph-client tries to connect to server with odd IP address In-Reply-To: <3B7A7BDFFD286F49A19A5B35D26E08FB8F11D9@srvmail2.aurenz.local> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8F116A@srvmail2.aurenz.local> <87liqrtb5o.fsf@gmail.com> <3B7A7BDFFD286F49A19A5B35D26E08FB8F11AF@srvmail2.aurenz.local> <87fwgzt0gv.fsf@gmail.com> <3B7A7BDFFD286F49A19A5B35D26E08FB8F11D9@srvmail2.aurenz.local> Message-ID: <87vcptshwc.fsf@gmail.com> On Tue, 6 Dec 2011 09:02:38 +0100, "Hohl, Gerrit" wrote: > Good morning Dmitry, > > here is the current test case I'm using. With these values it works perfectly. > But if change the peak_req_rate to 25000/sec, the polygraph-server will start 567 servers while the polygraph-client starts 250 robots, expecting 1126 servers. > I guess it is not valid to have a bigger peak_req_rate than the > max_host_load. It is valid. There may be multiple hosts in the test. > But instead of using a false number of servers (shouldn't it be the > same for client as well as for the server?) maybe it should quit by > showing an error that the parameters are not valid? > These are valid parameters. The problem is that the system does not have all the needed aliases, rerun polygraph-aka(1) tool with the new address space values. Please see my answer in another thread for details [1]. Regards, Dmitry [1] http://www.web-polygraph.org/pipermail/users/2011-December/000148.html From g.hohl at aurenz.de Thu Dec 8 15:29:05 2011 From: g.hohl at aurenz.de (Hohl, Gerrit) Date: Thu, 8 Dec 2011 16:29:05 +0100 Subject: too many open files - a lack of available filedescriptors? Message-ID: <3B7A7BDFFD286F49A19A5B35D26E08FB8F13F4@srvmail2.aurenz.local> Hello everyone, it is again me. :-P Now everything is working with 2 clients with totally 1250 robots. I have 2 servers, each with 375 aliases (750 servers). The server side is working perfectly, but on the two client machines I get the message that too many files are open. I assumed that I have to less file descripters available for them. But I looked it up: root at polygraph-client1:/root/polygraph# more /proc/sys/fs/file-nr 416 0 98945 root at polygraph-client1:/root/polygraph# lsof | wc -l 837 98945 - 837 = 98108 still available. Don't get it why that number is sufficient for 625 robots on each machine? Maybe I should have a look on the number while executing the test case... Okay, I tested it. At this point I get the message the first time: root at polygraph-client1:~# more /proc/sys/fs/file-nr ; lsof | wc -l 1504 0 98945 1995 1995 of 98945 file descriptors used. Do I look at the wrong values? :-? Regards, Gerrit From dwm at xpasc.com Thu Dec 8 16:28:38 2011 From: dwm at xpasc.com (David Morris) Date: Thu, 8 Dec 2011 08:28:38 -0800 (PST) Subject: too many open files - a lack of available filedescriptors? In-Reply-To: <3B7A7BDFFD286F49A19A5B35D26E08FB8F13F4@srvmail2.aurenz.local> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8F13F4@srvmail2.aurenz.local> Message-ID: You are looking a system wide values. You need the value for your polygraph process(es) ... 1024 is a typical default if you've not taken steps to increase it. On Thu, 8 Dec 2011, Hohl, Gerrit wrote: > Hello everyone, > > it is again me. :-P > Now everything is working with 2 clients with totally 1250 robots. I > have 2 servers, each with 375 aliases (750 servers). > The server side is working perfectly, but on the two client machines I > get the message that too many files are open. I assumed that I have to > less file descripters available for them. But I looked it up: > > root at polygraph-client1:/root/polygraph# more /proc/sys/fs/file-nr > 416 0 98945 > root at polygraph-client1:/root/polygraph# lsof | wc -l > 837 > > 98945 - 837 = 98108 still available. Don't get it why that number is > sufficient for 625 robots on each machine? > > Maybe I should have a look on the number while executing the test > case... > > > > Okay, I tested it. At this point I get the message the first time: > > root at polygraph-client1:~# more /proc/sys/fs/file-nr ; lsof | wc -l > 1504 0 98945 > 1995 > > 1995 of 98945 file descriptors used. Do I look at the wrong values? :-? > > Regards, > Gerrit > _______________________________________________ > Users mailing list > Users at web-polygraph.org > http://www.web-polygraph.org/mailman/listinfo/users > From chengy.fan at gmail.com Thu Dec 8 22:17:10 2011 From: chengy.fan at gmail.com (Chengyu Fan) Date: Thu, 8 Dec 2011 15:17:10 -0700 Subject: A couple of questions about report Message-ID: Hi~ all, Sorry I have some naive questions again. I modify the simple download test in my experiment. And I generate the report using polygraph-reporter. I am a little confused about some parts in the report. At the end of this test, the polygraph client cannot connect to the server side, and exits. The generated report is attached. The questions are as follows: 1. The load trace graph shows the offered rate and measured rate, but it only based on the client-side. So the client cannot distinguish the measured rate comes from the proxy or the polygraph server, right? Is there a graph we can directly check how many rate comes from proxy and how many comes from the server? 2. In hit ratios part, I find the measured number in the first table is more than 79% (I set the recurrence is 80%). But in the client side hit ratios, the measured number is 0. Why is that? 3. Also, I find the server side has lots of errors like "foreign host name" and "missing target info in the request", does that mean the proxy does not work well? Or this is the polygraph deliberately to do so? 4. I am also do not know what's the meaning of fill. Could you please explain it? Thanks. -- Regards, Chengyu Fan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: report.tar Type: application/x-tar Size: 322978 bytes Desc: not available URL: From g.hohl at aurenz.de Fri Dec 9 08:06:52 2011 From: g.hohl at aurenz.de (Hohl, Gerrit) Date: Fri, 9 Dec 2011 09:06:52 +0100 Subject: AW: too many open files - a lack of available filedescriptors? In-Reply-To: References: <3B7A7BDFFD286F49A19A5B35D26E08FB8F13F4@srvmail2.aurenz.local> Message-ID: <3B7A7BDFFD286F49A19A5B35D26E08FB8F1407@srvmail2.aurenz.local> Hello David, thanks for your reply. :-) I didn't know that there is a difference between the global and the per process limit. And I guess I found a solution here: http://www.xenoclast.org/doc/benchmark/HTTP-benchmarking-HOWTO/node7.html Regards, Gerrit -----Urspr?ngliche Nachricht----- Von: users-bounces at web-polygraph.org [mailto:users-bounces at web-polygraph.org] Im Auftrag von David Morris Gesendet: Donnerstag, 8. Dezember 2011 17:29 An: users at web-polygraph.org Betreff: Re: too many open files - a lack of available filedescriptors? You are looking a system wide values. You need the value for your polygraph process(es) ... 1024 is a typical default if you've not taken steps to increase it. On Thu, 8 Dec 2011, Hohl, Gerrit wrote: > Hello everyone, > > it is again me. :-P > Now everything is working with 2 clients with totally 1250 robots. I > have 2 servers, each with 375 aliases (750 servers). > The server side is working perfectly, but on the two client machines I > get the message that too many files are open. I assumed that I have to > less file descripters available for them. But I looked it up: > > root at polygraph-client1:/root/polygraph# more /proc/sys/fs/file-nr > 416 0 98945 > root at polygraph-client1:/root/polygraph# lsof | wc -l > 837 > > 98945 - 837 = 98108 still available. Don't get it why that number is > sufficient for 625 robots on each machine? > > Maybe I should have a look on the number while executing the test > case... > > > > Okay, I tested it. At this point I get the message the first time: > > root at polygraph-client1:~# more /proc/sys/fs/file-nr ; lsof | wc -l > 1504 0 98945 > 1995 > > 1995 of 98945 file descriptors used. Do I look at the wrong values? :-? > > Regards, > Gerrit > _______________________________________________ > Users mailing list > Users at web-polygraph.org > http://www.web-polygraph.org/mailman/listinfo/users > _______________________________________________ Users mailing list Users at web-polygraph.org http://www.web-polygraph.org/mailman/listinfo/users From dmitry.kurochkin at measurement-factory.com Fri Dec 9 14:11:56 2011 From: dmitry.kurochkin at measurement-factory.com (Dmitry Kurochkin) Date: Fri, 09 Dec 2011 18:11:56 +0400 Subject: A couple of questions about report In-Reply-To: References: Message-ID: <8762hp96mr.fsf@gmail.com> Hi Chengyu. On Thu, 8 Dec 2011 15:17:10 -0700, Chengyu Fan wrote: > Hi~ all, > Sorry I have some naive questions again. > I modify the simple download test in my experiment. And I generate the > report using polygraph-reporter. > I am a little confused about some parts in the report. At the end of this > test, the polygraph client cannot connect to the server side, and exits. > The generated report is attached. The questions are as follows: > > 1. The load trace graph shows the offered rate and measured rate, but it > only based on the client-side. If you mean the load trace figure on the "traffic rates, counts, and volumes" page, then yes. > So the client cannot distinguish the > measured rate comes from the proxy or the polygraph server, right? It can. Otherwise Polygraph would not be able to calculate Hit ratio. > Is there > a graph we can directly check how many rate comes from proxy and how many > comes from the server? > If proxy is used, all replies the client receives come from the proxy, clients do not talk directly to the origin servers in this case. Also I am not sure I understand what "how many rate" means. There are "traffic rates, counts, and volumes" page has client-side reply traffic stream table at the end. It contains contribution and rates stats for various reply types, including "misses" and "hits and misses". That may be what you are looking for. You can click on individual reply type to see detailed stats for it. Also the Details link on the index page goes to the index of all collected stats for all phases, both client and server sides. > 2. In hit ratios part, I find the measured number in the first table is > more than 79% (I set the recurrence is 80%). But in the client side hit > ratios, the measured number is 0. Why is that? > Polygraph relies on custom HTTP headers to calculate detailed hit stats. If for some reason Polygraph can not get these headers (e.g. proxy does not forward it), detailed hit ratio stats can not be calculated. Polygraph still can measure the average hit ratio based on client and server stats (the first table on the "hit ratios" page). > 3. Also, I find the server side has lots of errors like "foreign host name" > and "missing target info in the request", does that mean the proxy does not > work well? Or this is the polygraph deliberately to do so? > "Missing target info in the request" means that proxy did not forward X-Target request header. "Foreign host name" means that Polygraph server received request for an unknown host (either in the request URI or the Host header). Since you are using a single server, that means that proxy changed the host in some way. These issues are likely related to the missing client side stats problem. > 4. I am also do not know what's the meaning of fill. Could you please > explain it? > Fill is a cachable miss, i.e. a reply that may increase (fill) the proxy cache. Regards, Dmitry > Thanks. > > -- > Regards, > Chengyu Fan > _______________________________________________ > Users mailing list > Users at web-polygraph.org > http://www.web-polygraph.org/mailman/listinfo/users From chengy.fan at gmail.com Fri Dec 9 20:37:40 2011 From: chengy.fan at gmail.com (Chengyu Fan) Date: Fri, 9 Dec 2011 13:37:40 -0700 Subject: A couple of questions about report In-Reply-To: <8762hp96mr.fsf@gmail.com> References: <8762hp96mr.fsf@gmail.com> Message-ID: Thanks for reply so soon. On Fri, Dec 9, 2011 at 7:11 AM, Dmitry Kurochkin < dmitry.kurochkin at measurement-factory.com> wrote: > Hi Chengyu. > > On Thu, 8 Dec 2011 15:17:10 -0700, Chengyu Fan > wrote: > > Hi~ all, > > Sorry I have some naive questions again. > > I modify the simple download test in my experiment. And I generate the > > report using polygraph-reporter. > > I am a little confused about some parts in the report. At the end of this > > test, the polygraph client cannot connect to the server side, and exits. > > The generated report is attached. The questions are as follows: > > > > 1. The load trace graph shows the offered rate and measured rate, but it > > only based on the client-side. > > If you mean the load trace figure on the "traffic rates, counts, and > volumes" page, then yes. > > > So the client cannot distinguish the > > measured rate comes from the proxy or the polygraph server, right? > > It can. Otherwise Polygraph would not be able to calculate Hit ratio. > > > Is there > > a graph we can directly check how many rate comes from proxy and how many > > comes from the server? > > > > If proxy is used, all replies the client receives come from the proxy, > clients do not talk directly to the origin servers in this case. Also I > am not sure I understand what "how many rate" means. > For "how many rates", I mean right now we know the measured rate from the client side. But I don't know among all these measured rate, how many measured rate comes from server, and how many comes from proxy. Is there a way to do that? Also, I noticed that in the load trace, the measured rate is not always there. Sometimes I can only see the offered information. Why is this? > > There are "traffic rates, counts, and volumes" page has client-side > reply traffic stream table at the end. It contains contribution and > rates stats for various reply types, including "misses" and "hits and > misses". That may be what you are looking for. You can click on > individual reply type to see detailed stats for it. > > Also the Details link on the index page goes to the index of all > collected stats for all phases, both client and server sides. > Yeah, I checked that. But in the traffic stream table, I found that the misses is always 100%. But I do use a proxy, does this mean that the proxy does not cache anything? And I checked log file, the server side rate is much small than client sends. I think it is the proxy replies requests instead of server. So why the misses is 100%? > > > 2. In hit ratios part, I find the measured number in the first table is > > more than 79% (I set the recurrence is 80%). But in the client side hit > > ratios, the measured number is 0. Why is that? > > > > Polygraph relies on custom HTTP headers to calculate detailed hit > stats. If for some reason Polygraph can not get these headers > (e.g. proxy does not forward it), detailed hit ratio stats can not be > calculated. > > Polygraph still can measure the average hit ratio based on client and > server stats (the first table on the "hit ratios" page). > > > 3. Also, I find the server side has lots of errors like "foreign host > name" > > and "missing target info in the request", does that mean the proxy does > not > > work well? Or this is the polygraph deliberately to do so? > > > > "Missing target info in the request" means that proxy did not forward > X-Target request header. > > "Foreign host name" means that Polygraph server received request for an > unknown host (either in the request URI or the Host header). Since you > are using a single server, that means that proxy changed the host in > some way. > > These issues are likely related to the missing client side stats > problem. > Do you mean, "foreign host name" and "missing target info in the request" may be the reasons that there are such a high misses and no enough measured information in my report? > > > 4. I am also do not know what's the meaning of fill. Could you please > > explain it? > > > > Fill is a cachable miss, i.e. a reply that may increase (fill) the proxy > cache. > > Regards, > Dmitry > > > Thanks. > > > > -- > > Regards, > > Chengyu Fan > > _______________________________________________ > > Users mailing list > > Users at web-polygraph.org > > http://www.web-polygraph.org/mailman/listinfo/users > -- Regards, Chengyu Fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitry.kurochkin at measurement-factory.com Fri Dec 9 21:06:06 2011 From: dmitry.kurochkin at measurement-factory.com (Dmitry Kurochkin) Date: Sat, 10 Dec 2011 01:06:06 +0400 Subject: A couple of questions about report In-Reply-To: References: <8762hp96mr.fsf@gmail.com> Message-ID: <87mxb178w1.fsf@gmail.com> On Fri, 9 Dec 2011 13:37:40 -0700, Chengyu Fan wrote: > Thanks for reply so soon. > > On Fri, Dec 9, 2011 at 7:11 AM, Dmitry Kurochkin < > dmitry.kurochkin at measurement-factory.com> wrote: > > > Hi Chengyu. > > > > On Thu, 8 Dec 2011 15:17:10 -0700, Chengyu Fan > > wrote: > > > Hi~ all, > > > Sorry I have some naive questions again. > > > I modify the simple download test in my experiment. And I generate the > > > report using polygraph-reporter. > > > I am a little confused about some parts in the report. At the end of this > > > test, the polygraph client cannot connect to the server side, and exits. > > > The generated report is attached. The questions are as follows: > > > > > > 1. The load trace graph shows the offered rate and measured rate, but it > > > only based on the client-side. > > > > If you mean the load trace figure on the "traffic rates, counts, and > > volumes" page, then yes. > > > > > So the client cannot distinguish the > > > measured rate comes from the proxy or the polygraph server, right? > > > > It can. Otherwise Polygraph would not be able to calculate Hit ratio. > > > > > Is there > > > a graph we can directly check how many rate comes from proxy and how many > > > comes from the server? > > > > > > > If proxy is used, all replies the client receives come from the proxy, > > clients do not talk directly to the origin servers in this case. Also I > > am not sure I understand what "how many rate" means. > > > > For "how many rates", I mean right now we know the measured rate from the > client side. But I don't know among all these measured rate, how many > measured rate comes from server, and how many comes from proxy. > Is there a way to do that? > To collect detailed hits and misses stats, Polygraph needs the proxy to forward custom headers (X-Xact at least) in requests and replies. > Also, I noticed that in the load trace, the measured rate is not always > there. Sometimes I can only see the offered information. Why is this? > Apparently, in your test the proxy removes extension headers from forwarded messages. That is why Polygraph can not collect detailed hits and misses stats. > > > > > There are "traffic rates, counts, and volumes" page has client-side > > reply traffic stream table at the end. It contains contribution and > > rates stats for various reply types, including "misses" and "hits and > > misses". That may be what you are looking for. You can click on > > individual reply type to see detailed stats for it. > > > > Also the Details link on the index page goes to the index of all > > collected stats for all phases, both client and server sides. > > > > Yeah, I checked that. But in the traffic stream table, I found that the > misses is always 100%. But I do use a proxy, does this mean that the proxy > does not cache anything? And I checked log file, the server side rate is > much small than client sends. I think it is the proxy replies requests > instead of server. So why the misses is 100%? > I tried to explain why this happens above. In your test the proxy does caching, you can see the aggregated hit ratio in the first table on the "hit ratios" page. But detailed hits and misses stats can not be collected. > > > > > > 2. In hit ratios part, I find the measured number in the first table is > > > more than 79% (I set the recurrence is 80%). But in the client side hit > > > ratios, the measured number is 0. Why is that? > > > > > > > Polygraph relies on custom HTTP headers to calculate detailed hit > > stats. If for some reason Polygraph can not get these headers > > (e.g. proxy does not forward it), detailed hit ratio stats can not be > > calculated. > > > > Polygraph still can measure the average hit ratio based on client and > > server stats (the first table on the "hit ratios" page). > > > > > 3. Also, I find the server side has lots of errors like "foreign host > > name" > > > and "missing target info in the request", does that mean the proxy does > > not > > > work well? Or this is the polygraph deliberately to do so? > > > > > > > "Missing target info in the request" means that proxy did not forward > > X-Target request header. > > > > "Foreign host name" means that Polygraph server received request for an > > unknown host (either in the request URI or the Host header). Since you > > are using a single server, that means that proxy changed the host in > > some way. > > > > These issues are likely related to the missing client side stats > > problem. > > > > Do you mean, "foreign host name" and "missing target info in the request" > may be the reasons that there are such a high misses and no enough measured > information in my report? > "Foreign host name" errors seem like a separate issue. I do not know why the proxy changes host in forwarded requests but it is weird. As for the "missing target info in the request" errors, they are not the reason, they are the consequence. The root cause is that the proxy strips extension headers from the forwarded messages. If you fix that, the errors would be gone and you would get detailed hits and misses stats. Regards, Dmitry > > > > > > 4. I am also do not know what's the meaning of fill. Could you please > > > explain it? > > > > > > > Fill is a cachable miss, i.e. a reply that may increase (fill) the proxy > > cache. > > > > Regards, > > Dmitry > > > > > Thanks. > > > > > > -- > > > Regards, > > > Chengyu Fan > > > _______________________________________________ > > > Users mailing list > > > Users at web-polygraph.org > > > http://www.web-polygraph.org/mailman/listinfo/users > > > > > > -- > Regards, > Chengyu Fan From chengy.fan at gmail.com Fri Dec 9 21:31:48 2011 From: chengy.fan at gmail.com (Chengyu Fan) Date: Fri, 9 Dec 2011 14:31:48 -0700 Subject: A couple of questions about report In-Reply-To: <87mxb178w1.fsf@gmail.com> References: <8762hp96mr.fsf@gmail.com> <87mxb178w1.fsf@gmail.com> Message-ID: Thanks a lot! On Fri, Dec 9, 2011 at 2:06 PM, Dmitry Kurochkin < dmitry.kurochkin at measurement-factory.com> wrote: > On Fri, 9 Dec 2011 13:37:40 -0700, Chengyu Fan > wrote: > > Thanks for reply so soon. > > > > On Fri, Dec 9, 2011 at 7:11 AM, Dmitry Kurochkin < > > dmitry.kurochkin at measurement-factory.com> wrote: > > > > > Hi Chengyu. > > > > > > On Thu, 8 Dec 2011 15:17:10 -0700, Chengyu Fan > > > wrote: > > > > Hi~ all, > > > > Sorry I have some naive questions again. > > > > I modify the simple download test in my experiment. And I generate > the > > > > report using polygraph-reporter. > > > > I am a little confused about some parts in the report. At the end of > this > > > > test, the polygraph client cannot connect to the server side, and > exits. > > > > The generated report is attached. The questions are as follows: > > > > > > > > 1. The load trace graph shows the offered rate and measured rate, > but it > > > > only based on the client-side. > > > > > > If you mean the load trace figure on the "traffic rates, counts, and > > > volumes" page, then yes. > > > > > > > So the client cannot distinguish the > > > > measured rate comes from the proxy or the polygraph server, right? > > > > > > It can. Otherwise Polygraph would not be able to calculate Hit ratio. > > > > > > > Is there > > > > a graph we can directly check how many rate comes from proxy and how > many > > > > comes from the server? > > > > > > > > > > If proxy is used, all replies the client receives come from the proxy, > > > clients do not talk directly to the origin servers in this case. Also > I > > > am not sure I understand what "how many rate" means. > > > > > > > For "how many rates", I mean right now we know the measured rate from the > > client side. But I don't know among all these measured rate, how many > > measured rate comes from server, and how many comes from proxy. > > Is there a way to do that? > > > > To collect detailed hits and misses stats, Polygraph needs the proxy to > forward custom headers (X-Xact at least) in requests and replies. > > > Also, I noticed that in the load trace, the measured rate is not always > > there. Sometimes I can only see the offered information. Why is this? > > > > Apparently, in your test the proxy removes extension headers from > forwarded messages. That is why Polygraph can not collect detailed hits > and misses stats. > > > > > > > > > There are "traffic rates, counts, and volumes" page has client-side > > > reply traffic stream table at the end. It contains contribution and > > > rates stats for various reply types, including "misses" and "hits and > > > misses". That may be what you are looking for. You can click on > > > individual reply type to see detailed stats for it. > > > > > > Also the Details link on the index page goes to the index of all > > > collected stats for all phases, both client and server sides. > > > > > > > Yeah, I checked that. But in the traffic stream table, I found that the > > misses is always 100%. But I do use a proxy, does this mean that the > proxy > > does not cache anything? And I checked log file, the server side rate is > > much small than client sends. I think it is the proxy replies requests > > instead of server. So why the misses is 100%? > > > > I tried to explain why this happens above. In your test the proxy does > caching, you can see the aggregated hit ratio in the first table on the > "hit ratios" page. But detailed hits and misses stats can not be > collected. > > > > > > > > > > 2. In hit ratios part, I find the measured number in the first table > is > > > > more than 79% (I set the recurrence is 80%). But in the client side > hit > > > > ratios, the measured number is 0. Why is that? > > > > > > > > > > Polygraph relies on custom HTTP headers to calculate detailed hit > > > stats. If for some reason Polygraph can not get these headers > > > (e.g. proxy does not forward it), detailed hit ratio stats can not be > > > calculated. > > > > > > Polygraph still can measure the average hit ratio based on client and > > > server stats (the first table on the "hit ratios" page). > > > > > > > 3. Also, I find the server side has lots of errors like "foreign host > > > name" > > > > and "missing target info in the request", does that mean the proxy > does > > > not > > > > work well? Or this is the polygraph deliberately to do so? > > > > > > > > > > "Missing target info in the request" means that proxy did not forward > > > X-Target request header. > > > > > > "Foreign host name" means that Polygraph server received request for an > > > unknown host (either in the request URI or the Host header). Since you > > > are using a single server, that means that proxy changed the host in > > > some way. > > > > > > These issues are likely related to the missing client side stats > > > problem. > > > > > > > Do you mean, "foreign host name" and "missing target info in the request" > > may be the reasons that there are such a high misses and no enough > measured > > information in my report? > > > > "Foreign host name" errors seem like a separate issue. I do not know > why the proxy changes host in forwarded requests but it is weird. > > As for the "missing target info in the request" errors, they are not the > reason, they are the consequence. The root cause is that the proxy > strips extension headers from the forwarded messages. If you fix that, > the errors would be gone and you would get detailed hits and misses > stats. > > Regards, > Dmitry > > > > > > > > > > 4. I am also do not know what's the meaning of fill. Could you please > > > > explain it? > > > > > > > > > > Fill is a cachable miss, i.e. a reply that may increase (fill) the > proxy > > > cache. > > > > > > Regards, > > > Dmitry > > > > > > > Thanks. > > > > > > > > -- > > > > Regards, > > > > Chengyu Fan > > > > _______________________________________________ > > > > Users mailing list > > > > Users at web-polygraph.org > > > > http://www.web-polygraph.org/mailman/listinfo/users > > > > > > > > > > > -- > > Regards, > > Chengyu Fan > -- Regards, Chengyu Fan -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.hohl at aurenz.de Thu Dec 15 16:07:48 2011 From: g.hohl at aurenz.de (Hohl, Gerrit) Date: Thu, 15 Dec 2011 17:07:48 +0100 Subject: Problems using robots with authentication Message-ID: <3B7A7BDFFD286F49A19A5B35D26E08FB8F1860@srvmail2.aurenz.local> Hello everyone, I have the same problem that ufa faced at the end of October: I use NTLM authentication and get the message NtlmAuth.cc:798: assertion failed: 'false' Aborted from the polygraph-client program. After the first time I received that message I included the following line in my PGL file: Robot robot = { [...] pconn_use_lmt = const(2147483647); [...] }; But I still get the same message. I read something in Dmitry response about OpenSSL. The OpenSSL package ('openssl') is installed on that Ubuntu machine I'm currently using. But I'm not sure if polygraph was compiled with or without SSL. Is there a way to test it? Or what do I have to do to make sure that polygraph compiles with SSL support? Regards, Gerrit From g.hohl at aurenz.de Thu Dec 15 16:36:34 2011 From: g.hohl at aurenz.de (Hohl, Gerrit) Date: Thu, 15 Dec 2011 17:36:34 +0100 Subject: AW: Problems using robots with authentication In-Reply-To: <3B7A7BDFFD286F49A19A5B35D26E08FB8F1860@srvmail2.aurenz.local> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8F1860@srvmail2.aurenz.local> Message-ID: <3B7A7BDFFD286F49A19A5B35D26E08FB8F1868@srvmail2.aurenz.local> Hello everyone, I read the article "Prerequisites" in the documentation: http://www.web-polygraph.org/docs/reference/models/ssl.html#Sect:2 Polygraph SSL support is based on the OpenSSL library. A recent version of the library is required to compile Polygraph. We have tested with OpenSSL versions 0.9.6g and 0.9.7b. The presence of OpenSSL is determined at ./configure time. Please check that ./configure actually found SSL library and headers if you install Polygraph and want SSL support: ... checking for CRYPTO_lock in -lcrypto... yes checking for SSL_connect in -lssl... yes checking for openssl/ssl.h... yes checking for openssl/err.h... yes checking for openssl/rand.h... yes Eh, I don't get these lines when I call the script. I assume that the script was modified, but the documentation not. One remark: I only want NTLM authentication and not HTTPS benchmarking. Regards, Gerrit -----Urspr?ngliche Nachricht----- Von: users-bounces at web-polygraph.org [mailto:users-bounces at web-polygraph.org] Im Auftrag von Hohl, Gerrit Gesendet: Donnerstag, 15. Dezember 2011 17:08 An: users at web-polygraph.org Betreff: Problems using robots with authentication Hello everyone, I have the same problem that ufa faced at the end of October: I use NTLM authentication and get the message NtlmAuth.cc:798: assertion failed: 'false' Aborted from the polygraph-client program. After the first time I received that message I included the following line in my PGL file: Robot robot = { [...] pconn_use_lmt = const(2147483647); [...] }; But I still get the same message. I read something in Dmitry response about OpenSSL. The OpenSSL package ('openssl') is installed on that Ubuntu machine I'm currently using. But I'm not sure if polygraph was compiled with or without SSL. Is there a way to test it? Or what do I have to do to make sure that polygraph compiles with SSL support? Regards, Gerrit _______________________________________________ Users mailing list Users at web-polygraph.org http://www.web-polygraph.org/mailman/listinfo/users From dmitry.kurochkin at measurement-factory.com Thu Dec 15 20:11:47 2011 From: dmitry.kurochkin at measurement-factory.com (Dmitry Kurochkin) Date: Fri, 16 Dec 2011 00:11:47 +0400 Subject: AW: Problems using robots with authentication In-Reply-To: <3B7A7BDFFD286F49A19A5B35D26E08FB8F1868@srvmail2.aurenz.local> References: <3B7A7BDFFD286F49A19A5B35D26E08FB8F1860@srvmail2.aurenz.local> <3B7A7BDFFD286F49A19A5B35D26E08FB8F1868@srvmail2.aurenz.local> Message-ID: <8762hhtx18.fsf@gmail.com> Hi Gerrit. On Thu, 15 Dec 2011 17:36:34 +0100, "Hohl, Gerrit" wrote: > Hello everyone, > > I read the article "Prerequisites" in the documentation: > http://www.web-polygraph.org/docs/reference/models/ssl.html#Sect:2 > > Polygraph SSL support is based on the OpenSSL library. A recent version of the library is required to compile Polygraph. We have tested with OpenSSL versions 0.9.6g and 0.9.7b. The presence of OpenSSL is determined at ./configure time. Please check that ./configure actually found SSL library and headers if you install Polygraph and want SSL support: > > ... > checking for CRYPTO_lock in -lcrypto... yes > checking for SSL_connect in -lssl... yes > checking for openssl/ssl.h... yes > checking for openssl/err.h... yes > checking for openssl/rand.h... yes > > Eh, I don't get these lines when I call the script. I assume that the script was modified, but the documentation not. > The exact messages may have change, but they are still there: $ ./configure | grep -i ssl checking for SSL_connect in -lssl... yes checking openssl/ssl.h usability... yes checking openssl/ssl.h presence... yes checking for openssl/ssl.h... yes checking openssl/err.h usability... yes checking openssl/err.h presence... yes checking for openssl/err.h... yes checking openssl/rand.h usability... yes checking openssl/rand.h presence... yes checking for openssl/rand.h... yes Assertion at NtlmAuth.cc:798 means that you built without OpenSSL support. You should install it and rebuild. On Debian-based systems you should install libssl-dev package. The assertion is always a bug. Web Polygraph should print a proper error here. There is an open bug #878881 [1] for this issue. > One remark: I only want NTLM authentication and not HTTPS benchmarking. > NTLM needs some crypto functions (MD5, at least). That is why OpenSSL is needed for it. Regards, Dmitry [1] https://bugs.launchpad.net/polygraph/+bug/878881 > Regards, > Gerrit > > -----Urspr?ngliche Nachricht----- > Von: users-bounces at web-polygraph.org [mailto:users-bounces at web-polygraph.org] Im Auftrag von Hohl, Gerrit > Gesendet: Donnerstag, 15. Dezember 2011 17:08 > An: users at web-polygraph.org > Betreff: Problems using robots with authentication > > Hello everyone, > > I have the same problem that ufa faced at the end of October: > I use NTLM authentication and get the message > > NtlmAuth.cc:798: assertion failed: 'false' > Aborted > > from the polygraph-client program. After the first time I received that message I included the following line in my PGL file: > > Robot robot = { > [...] > pconn_use_lmt = const(2147483647); > [...] > }; > > But I still get the same message. I read something in Dmitry response about OpenSSL. The OpenSSL package ('openssl') is installed on that Ubuntu machine I'm currently using. But I'm not sure if polygraph was compiled with or without SSL. Is there a way to test it? Or what do I have to do to make sure that polygraph compiles with SSL support? > > Regards, > Gerrit > _______________________________________________ > Users mailing list > Users at web-polygraph.org > http://www.web-polygraph.org/mailman/listinfo/users > _______________________________________________ > Users mailing list > Users at web-polygraph.org > http://www.web-polygraph.org/mailman/listinfo/users From caleb.harris at webhostgear.com Fri Dec 23 11:41:01 2011 From: caleb.harris at webhostgear.com (Caleb Harris) Date: Fri, 23 Dec 2011 12:41:01 +0100 Subject: interested in Web-Polygraph for our site Message-ID: <43da643bc984f8364413c3f1826c3446@localhost.localdomain> Hi Web-Polygraph team, Our website offers a directory which list sites that offer commendable web development tools as an assistance to our visitors. We are currently looking for sites to be a part of it. I received feedback with good compliments for Web-Polygraph by my site's readers. They love your tool's usability, flexibility, efficiency and effectiveness. Would you like to hear the details? Best Regards, Caleb Harris