From alberto.klocker at bluereef.com.au Wed Mar 5 04:05:12 2014 From: alberto.klocker at bluereef.com.au (Alberto Klocker) Date: Wed, 5 Mar 2014 15:05:12 +1100 Subject: Running multiple SSL servers on same machine Message-ID: Hi, I've been using tasksets as recommended to run multiple instances of the polygraph-server on a single machine with great results. However when it comes to using SSL on the servers I seem to be hitting a wall. Some of my server processes with SSL peak at 100% CPU usage under high load tests. If I use tasksets to spread out across cores I find I start getting errors because when starting up they create the .pem files in /tmp/ with the same names. ie all servers create files in /tmp named poly-1b5ac501-cert.srl,poly-557ac53b-serverkey.pem poly-6401b68e-serverkey.pem etc Initially they all used the same myssl.conf file from your example in the docs. I tried placing these in different directories with different configurations but the files seem to always have the same name. Any thoughts on how I can get multiple SSL servers running on one machine? *Alberto Klocker* | Software Developer | www.bluereef.com.au *_________________________________________________________________* *D: *+61 3 9895 8006 | *T*: +61 3 9898 8000 | @AlbertoKlocker Please consider the environment before printing this email -------------- next part -------------- An HTML attachment was scrubbed... URL: From rousskov at measurement-factory.com Fri Mar 7 21:45:13 2014 From: rousskov at measurement-factory.com (Alex Rousskov) Date: Fri, 07 Mar 2014 14:45:13 -0700 Subject: Running multiple SSL servers on same machine In-Reply-To: References: Message-ID: <531A3DE9.3000004@measurement-factory.com> On 03/04/2014 09:05 PM, Alberto Klocker wrote: > I've been using tasksets as recommended to run multiple instances of the > polygraph-server on a single machine with great results. However when it > comes to using SSL on the servers I seem to be hitting a wall. > > Some of my server processes with SSL peak at 100% CPU usage under high > load tests. If I use tasksets to spread out across cores I find I start > getting errors because when starting up they create the .pem files in > /tmp/ with the same names. > > ie all servers create files in /tmp > named poly-1b5ac501-cert.srl,poly-557ac53b-serverkey.pem > poly-6401b68e-serverkey.pem etc > > Initially they all used the same myssl.conf file from your example in > the docs. I tried placing these in different directories with different > configurations but the files seem to always have the same name. Any > thoughts on how I can get multiple SSL servers running on one machine? Yes, please try the attached patch that makes certificate file names more unique. We use it for SMP testing. HTH, Alex. -------------- next part -------------- A non-text attachment was scrubbed... Name: smp-cert-gen-t1.patch Type: text/x-diff Size: 738 bytes Desc: not available URL: From thealex.xu1986 at gmail.com Mon Mar 10 08:02:21 2014 From: thealex.xu1986 at gmail.com (xu tony) Date: Mon, 10 Mar 2014 16:02:21 +0800 Subject: a issue when running the "Simple tests" with a proxy Message-ID: I've been using the "Simple tests" introdued by polygraph's User manual to test my squid server. During testing, I found my squid can't pull the resource from orign(that is polygraph-server) successfully. The environment used in tests is as follows: 1?Two testing maches: one used for squid server with ip addr 218.202.111.5 and another used for polygraph-server and polygraph-client with ip addr 218.202.111.138.And my squid listens on the 8101 port. 2?The config file used by polygraph is as following: ------------------------------------------------------------------------------------------------------- Content SimpleContent = { size = exp(13KB); // response sizes distributed exponentially cachable = 80%; // 20% of content is uncachable }; Server S = { kind = "S101"; contents = [ SimpleContent ]; direct_access = contents; addresses = ['218.202.111.138:80']; // where to create these server agents }; // a primitive robot Robot R = { kind = "R101"; pop_model = { pop_distr = popUnif(); }; recurrence = 55% / SimpleContent.cachable; // adjusted to get 55% DHR req_rate = 1/sec; origins = S.addresses; // where the origin servers are addresses = ['218.202.111.138' ** 10]; // where these robot agents will be created }; // commit to using these servers and robots use(S, R); 3?The command i used for starting polygraph-server and polygraph-client is , /usr/local/polygraph/bin/polygraph-client --config ./my_simple.pg--verb_lvl 10 /usr/local/polygraph/bin/polygraph-client --config ../workdir/my_simple.pg--verb_lvl 10 --proxy 218.202.111.5:8101 During test, i found squid have got requests from ploygraph-client,but using a wrong url to pull resource (that is polygraph-server).Part of my squid access log is showed as below: -------------------------------------------------------------------------------------------------------------- 218.202.111.138 - - [10/Mar/2014:15:58:49 +0800] "GET http://w1cbc2601.7f1c694f:8/t03/_00000003 HTTP/1.1" 503 1513 "-" "-" TCP_MISS DIRECT 0 - 218.202.111.138 - - [10/Mar/2014:15:58:50 +0800] "GET http://w1cbc2601.7f1c694f:8/t03/_00000004 HTTP/1.1" 503 1513 "-" "-" TCP_MISS DIRECT 0 - I snapped http-request packets from polygraph-client, and found request-line of http-request is not what squid expected,which caused the squid can't pull resource from polyserver.The request-line in http-request sent by polyclient is like that : "GET http://218.202.111.138/w1cbc0feb.0fea1bae:00000008/t03/_00000001HTTP/1.1" The right request-line which the squid expected ,i think, shoud be like: "GET /218.202.111.138/w1cbc02ee.0c726ba0:00000008/t03/_00000015 HTTP/1.0" So, i want to know which part is wrong in my test, if i have made some mistakes in test, or polyclient sent http-request with wrong request-line, or my squid had some problems ,or anything else? How can i fix it? Any idea?please advice, and many thanks in advanced!!! -------------- next part -------------- An HTML attachment was scrubbed... URL: From pavel.kazlenka at measurement-factory.com Mon Mar 10 15:57:47 2014 From: pavel.kazlenka at measurement-factory.com (Pavel Kazlenka) Date: Mon, 10 Mar 2014 18:57:47 +0300 Subject: a issue when running the "Simple tests" with a proxy In-Reply-To: References: Message-ID: <531DE0FB.4050007@measurement-factory.com> Hi xu, Seems like you've started two polygraph-clients instead of one client and one server: >/usr/local/polygraph/bin/polygraph-client --config ./my_simple.pg --verb_lvl 10 >/usr/local/polygraph/bin/polygraph-client --config ../workdir/my_simple.pg --verb_lvl 10 --proxy 218.202.111.5:8101 Best wishes, Pavel On 03/10/2014 11:02 AM, xu tony wrote: > I've been using the "Simple tests" introdued by polygraph's User > manual to test my squid server. During testing, I found my squid > can't pull the resource from orign(that is polygraph-server) > successfully. > > The environment used in tests is as follows: > 1?Two testing maches: one used for squid server with ip addr > 218.202.111.5 and another used for polygraph-server and > polygraph-client with ip addr 218.202.111.138.And my squid listens on > the 8101 port. > 2?The config file used by polygraph is as following: > ------------------------------------------------------------------------------------------------------- > Content SimpleContent = { > size = exp(13KB); // response sizes distributed exponentially > cachable = 80%; // 20% of content is uncachable > }; > Server S = { > kind = "S101"; > contents = [ SimpleContent ]; > direct_access = contents; > > addresses = ['218.202.111.138:80']; // where to create these > server agents > }; > // a primitive robot > Robot R = { > kind = "R101"; > pop_model = { pop_distr = popUnif(); }; > recurrence = 55% / SimpleContent.cachable; // adjusted to get > 55% DHR > > req_rate = 1/sec; > origins = S.addresses; // where the origin servers are > addresses = ['218.202.111.138' ** 10]; // where these robot > agents will be created > }; > // commit to using these servers and robots > use(S, R); > 3?The command i used for starting polygraph-server and > polygraph-client is , > /usr/local/polygraph/bin/polygraph-client --config ./my_simple.pg > --verb_lvl 10 > /usr/local/polygraph/bin/polygraph-client --config > ../workdir/my_simple.pg --verb_lvl 10 --proxy > 218.202.111.5:8101 > > During test, i found squid have got requests from > ploygraph-client,but using a wrong url to pull resource (that is > polygraph-server).Part of my squid access log is showed as below: > -------------------------------------------------------------------------------------------------------------- > 218.202.111.138 - - [10/Mar/2014:15:58:49 +0800] "GET > http://w1cbc2601.7f1c694f:8/t03/_00000003 HTTP/1.1" 503 1513 "-" "-" > TCP_MISS DIRECT 0 - > 218.202.111.138 - - [10/Mar/2014:15:58:50 +0800] "GET > http://w1cbc2601.7f1c694f:8/t03/_00000004 HTTP/1.1" 503 1513 "-" "-" > TCP_MISS DIRECT 0 - > I snapped http-request packets from polygraph-client, and found > request-line of http-request is not what squid expected,which caused > the squid can't pull resource from polyserver.The request-line in > http-request sent by polyclient is like that : > "GET http://218.202.111.138/w1cbc0feb.0fea1bae:00000008/t03/_00000001 > HTTP/1.1" > The right request-line which the squid expected ,i think, shoud be like: > "GET /218.202.111.138/w1cbc02ee.0c726ba0:00000008/t03/_00000015 > > HTTP/1.0" > > So, i want to know which part is wrong in my test, if i have made > some mistakes in test, or polyclient sent http-request with wrong > request-line, or my squid had some problems ,or anything else? How can > i fix it? Any idea?please advice, and many thanks in advanced!!! > > > _______________________________________________ > Users mailing list > Users at web-polygraph.org > http://www.web-polygraph.org/mailman/listinfo/users -------------- next part -------------- An HTML attachment was scrubbed... URL: From rousskov at measurement-factory.com Mon Mar 10 16:05:54 2014 From: rousskov at measurement-factory.com (Alex Rousskov) Date: Mon, 10 Mar 2014 10:05:54 -0600 Subject: a issue when running the "Simple tests" with a proxy In-Reply-To: References: Message-ID: <531DE2E2.5030502@measurement-factory.com> On 03/10/2014 02:02 AM, xu tony wrote: > I've been using the "Simple tests" introdued by polygraph's User > manual to test my squid server. During testing, I found my squid can't > pull the resource from orign(that is polygraph-server) successfully. > 3?The command i used for starting polygraph-server and polygraph-client > is , > /usr/local/polygraph/bin/polygraph-client --config > ../workdir/my_simple.pg --verb_lvl 10 --proxy > 218.202.111.5:8101 > During test, i found squid have got requests from > ploygraph-client,but using a wrong url to pull resource (that is > polygraph-server).Part of my squid access log is showed as below: > -------------------------------------------------------------------------------------------------------------- > 218.202.111.138 - - [10/Mar/2014:15:58:49 +0800] "GET > http://w1cbc2601.7f1c694f:8/t03/_00000003 HTTP/1.1" 503 1513 "-" "-" > TCP_MISS DIRECT 0 - The wrong domain name in the above log implies that your Squid is confused as to where the requests are going. This would not happen with a regular Squid configuration. Starting with a simple forward proxy configuration for Squid may be a good idea before moving on to more complex stuff (squid.conf.default may work after adding http_access rules to allow Polygraph traffic). > I snapped http-request packets from polygraph-client, and found > request-line of http-request is not what squid expected,which caused the > squid can't pull resource from polyserver.The request-line in > http-request sent by polyclient is like that : > "GET http://218.202.111.138/w1cbc0feb.0fea1bae:00000008/t03/_00000001 > HTTP/1.1" Yes, this is what you told Polygraph to send. In a forward proxy test that you are running (from Polygraph point of view), the URL should look like the one logged above. > The right request-line which the squid expected ,i think, shoud be like: > "GET /218.202.111.138/w1cbc02ee.0c726ba0:00000008/t03/_00000015 HTTP/1.0" No, but if you are testing a reverse proxy, the Request URI that Squid expects should look like this: GET /w1cbc02ee.0c726ba0:00000008/t03/_00000015 HTTP/... It would still be logged as a full URL (with a host name) by Squid IIRC, but the packet capture should show from-Polygraph Request URIs without the protocol scheme and host name components. Please note that some packet capture GUIs such as Wireshark sometimes "restore" the full URL when rendering high-level details of the captured packet, even when a full URL is not present in the packet. Make sure you look at what has been actually sent and what what was rendered by GUI. > So, i want to know which part is wrong in my test, if i have made > some mistakes in test, or polyclient sent http-request with wrong > request-line, or my squid had some problems ,or anything else? How can i > fix it? You have configured Polygraph for a forward proxy test (the --proxy option does that). One possibility is that you have also configured your Squid to be a reverse proxy. If the two configurations are out of sync, bad things like the one you have described may happen. If you want to test a reverse proxy, you have to change Polygraph configuration. Do not use --proxy at all (reverse proxies are origin servers from the HTTP protocol point of view). Add a PGL AddrMap that maps proxy address (218.202.111.5:8101) to the Polygraph server address (S.addresses). And use map.names for Robot's origins field (R.origins) instead of S.addresses you are using there now. The WebAxe workload configuration distributed with Polygraph has a [more complex] example on how to do that. If you do not want to test a reverse proxy, reconfigure your Squid to be a regular forward proxy (starting with squid.conf.default may be a good idea). HTH, Alex. From thealex.xu1986 at gmail.com Tue Mar 11 06:23:00 2014 From: thealex.xu1986 at gmail.com (xu alex) Date: Tue, 11 Mar 2014 14:23:00 +0800 Subject: a issue when running the "Simple tests" with a proxy In-Reply-To: <531DE2E2.5030502@measurement-factory.com> References: <531DE2E2.5030502@measurement-factory.com> Message-ID: Hi Alex? Thanks for your reply. Yes, I has been testing squid using as a reverse proxy. I tried the way of adding PGL AddrMap in my config file, but errors showed up when starting polygraph-client. My config file is as following: ---------------------------------------------------------------------------------------------------------------------------------- ObjLifeCycle olcStatic = { birthday = now + const(-1year); // born a year ago length = const(2year); // two year cycle variance = 0%; // no variance with_lmt = 100%; // all responses have LMT expires = [nmt + const(0sec)]; // everything expires when modified }; Content SimpleContent = { size = exp(13000KB); // response sizes distributed exponentially cachable = 100%; // 20% of content is uncachable obj_life_cycle = olcStatic; }; Server S = { kind = "S101"; contents = [ SimpleContent ]; direct_access = contents; addresses = ['218.202.111.138:80']; // where to create these server agents }; AddrMap Map = { addresses = S.addresses; names = ['218.202.111.5:8101']; }; // a primitive robot Robot R = { kind = "R101"; pop_model = { pop_distr = popUnif(); }; recurrence = 55% / SimpleContent.cachable; // adjusted to get 55% DHR req_rate = 10/sec; origins = Map.names; // where the origin servers are addresses = ['218.202.111.138' ** 10]; // where these robot agents will be created }; // commit to using these servers and robots use(S, R); The error info showed up when i started polygraph-clients is as following: ------------------------------------------------------------------------------------------------------------------------------------------------------ fyi: PGL configuration stored (664bytes) 000.00| fyi: all 10 Robot IP address(es) lack interface name or netmask 000.00| found 10 Robot IP address(es) total (0 unique address(es) with interface name and netmask). 000.00| fyi: no bench selected with use() 000.00| ./my_simple.pg:41: error: Robot cannot find configuration for server at 218.202.111.5:8101 I am a newer for polygraph, and can not unstand WebAxe workload configuration completely, so i think there are some mistakes in my configuration file. Can you help me take a look at my config file showed above for which part is wrong, and how i should modify it? Currently, I just modify the source code as following to make it work with my squid server: --------------------------------------------------------------------------------------------------------------------------------------- ostream &Oid2Url(const ObjId &oid, ostream &os) { if (oid.foreignUrl()) return os << oid.foreignUrl(); if (oid.secure()) os << "https://"; else - os << (oid.scheme() == Agent::pHTTP ? "http://" : "ftp://"); + os << (oid.scheme() == Agent::pHTTP ? "/" : "ftp://"); Oid2UrlHost(oid, false, os); Oid2UrlPath(oid, os); return os; } I think that is not right way to use polygraph for test reverse proxy, so look forward to your reply! 2014-03-11 0:05 GMT+08:00 Alex Rousskov : > On 03/10/2014 02:02 AM, xu tony wrote: > > I've been using the "Simple tests" introdued by polygraph's User > > manual to test my squid server. During testing, I found my squid can't > > pull the resource from orign(that is polygraph-server) successfully. > > > 3?The command i used for starting polygraph-server and polygraph-client > > is , > > /usr/local/polygraph/bin/polygraph-client --config > > ../workdir/my_simple.pg --verb_lvl 10 --proxy > > 218.202.111.5:8101 > > > > During test, i found squid have got requests from > > ploygraph-client,but using a wrong url to pull resource (that is > > polygraph-server).Part of my squid access log is showed as below: > > > -------------------------------------------------------------------------------------------------------------- > > 218.202.111.138 - - [10/Mar/2014:15:58:49 +0800] "GET > > http://w1cbc2601.7f1c694f:8/t03/_00000003 HTTP/1.1" 503 1513 "-" "-" > > TCP_MISS DIRECT 0 - > > The wrong domain name in the above log implies that your Squid is > confused as to where the requests are going. This would not happen with > a regular Squid configuration. Starting with a simple forward proxy > configuration for Squid may be a good idea before moving on to more > complex stuff (squid.conf.default may work after adding http_access > rules to allow Polygraph traffic). > > > > I snapped http-request packets from polygraph-client, and found > > request-line of http-request is not what squid expected,which caused the > > squid can't pull resource from polyserver.The request-line in > > http-request sent by polyclient is like that : > > "GET http://218.202.111.138/w1cbc0feb.0fea1bae:00000008/t03/_00000001 > > HTTP/1.1" > > Yes, this is what you told Polygraph to send. In a forward proxy test > that you are running (from Polygraph point of view), the URL should look > like the one logged above. > > > > The right request-line which the squid expected ,i think, shoud be like: > > "GET /218.202.111.138/w1cbc02ee.0c726ba0:00000008/t03/_00000015HTTP/1.0" > > No, but if you are testing a reverse proxy, the Request URI that Squid > expects should look like this: > > GET /w1cbc02ee.0c726ba0:00000008/t03/_00000015 HTTP/... > > It would still be logged as a full URL (with a host name) by Squid IIRC, > but the packet capture should show from-Polygraph Request URIs without > the protocol scheme and host name components. > > Please note that some packet capture GUIs such as Wireshark sometimes > "restore" the full URL when rendering high-level details of the captured > packet, even when a full URL is not present in the packet. Make sure you > look at what has been actually sent and what what was rendered by GUI. > > > > So, i want to know which part is wrong in my test, if i have made > > some mistakes in test, or polyclient sent http-request with wrong > > request-line, or my squid had some problems ,or anything else? How can i > > fix it? > > You have configured Polygraph for a forward proxy test (the --proxy > option does that). One possibility is that you have also configured your > Squid to be a reverse proxy. If the two configurations are out of sync, > bad things like the one you have described may happen. > > If you want to test a reverse proxy, you have to change Polygraph > configuration. Do not use --proxy at all (reverse proxies are origin > servers from the HTTP protocol point of view). Add a PGL AddrMap that > maps proxy address (218.202.111.5:8101) to the Polygraph server address > (S.addresses). And use map.names for Robot's origins field (R.origins) > instead of S.addresses you are using there now. The WebAxe workload > configuration distributed with Polygraph has a [more complex] example on > how to do that. > > If you do not want to test a reverse proxy, reconfigure your Squid to be > a regular forward proxy (starting with squid.conf.default may be a good > idea). > > > HTH, > > Alex. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rousskov at measurement-factory.com Tue Mar 11 16:15:37 2014 From: rousskov at measurement-factory.com (Alex Rousskov) Date: Tue, 11 Mar 2014 10:15:37 -0600 Subject: a issue when running the "Simple tests" with a proxy In-Reply-To: References: <531DE2E2.5030502@measurement-factory.com> Message-ID: <531F36A9.8050503@measurement-factory.com> On 03/11/2014 12:23 AM, xu alex wrote: > Thanks for your reply. Yes, I has been testing squid using as a > reverse proxy. I tried the way of adding PGL AddrMap in my config file, > but errors showed up when starting polygraph-client. My config file is > as following: > AddrMap Map = { > addresses = S.addresses; > names = ['218.202.111.5:8101']; > }; > The error info showed up when i started polygraph-clients is as following: > 000.00| ./my_simple.pg:41: error: Robot cannot > find configuration for server at 218.202.111.5:8101 You need to use() Map, just like you use() your Robot and Server. Folks may maintain a collections or libraries of maps (using some of those maps for some tests), and Polygraph does not currently try to guess which maps you meant to use. HTH, Alex.