urlopen returns empty string

urlopen returns empty string

Postby SEnergy » Sun Aug 21, 2016 2:55 am

I'm trying to make some API calls, however the code below returns empty string, unless the parameters are hardcoded into url

Code: Select all
#https://poloniex.com/public?command=returnLoanOrders&currency=BTC
params = {"command":"returnLoanOrders", "currency":"BTC"}
encoded = urllib.urlencode(params)

#this works
print urllib2.urlopen(urllib2.Request("https://poloniex.com/public?currency=BTC&command=returnLoanOrders")).read()
print urllib2.urlopen("https://poloniex.com/public?currency=BTC&command=returnLoanOrders").read()

#this does not
print urllib2.urlopen(urllib2.Request("https://poloniex.com/public", encoded)).read()
print urllib2.urlopen("https://poloniex.com/public", encoded).read()

print encoded


output:
{"offers":[{"rate":"0.00038499","amount":"0.00100000","rangeMin":2,"rangeMax":2},{"rate":"0.00038500","amount":"0.00202262","rangeMin":2,"rangeMax":2},{"rate":"0.00038600","amount":"0.97809855","rangeMin":2,"rangeMax":2},{"rate":"0.00038700","amount":"0.11781954","rangeMin":2,"rangeMax":2},{"rate":"0.00038800","amount":"0.00008255","rangeMin":2,"rangeMax":2},{"rate":"0.00038885","amount":"0.00383026","rangeMin":2,"rangeMax":2},{"rate":"0.00038900","amount":"0.00181875","rangeMin":2,"rangeMax":2},{"rate":"0.00039000","amount":"0.31843708","rangeMin":2,"rangeMax":3},{"rate":"0.00039200","amount":"0.03263231","rangeMin":2,"rangeMax":2},{"rate":"0.00039300","amount":"0.02604170","rangeMin":2,"rangeMax":2},{"rate":"0.00039400","amount":"0.06210503","rangeMin":2,"rangeMax":2},{"rate":"0.00039900","amount":"0.02034900","rangeMin":2,"rangeMax":2},{"rate":"0.00040000","amount":"0.14834252","rangeMin":2,"rangeMax":2},{"rate":"0.00040198","amount":"0.00174736","rangeMin":2,"rangeMax":2},{"rate":"0.00040600","amount":"0.01058767","rangeMin":2,"rangeMax":2},{"rate":"0.00040900","amount":"0.02221216","rangeMin":2,"rangeMax":2},{"rate":"0.00043175","amount":"0.01399529","rangeMin":2,"rangeMax":2},{"rate":"0.00044000","amount":"0.62522560","rangeMin":2,"rangeMax":2},{"rate":"0.00045100","amount":"0.03316436","rangeMin":2,"rangeMax":2},{"rate":"0.00045199","amount":"0.01366751","rangeMin":2,"rangeMax":2},{"rate":"0.00045200","amount":"3.40728067","rangeMin":2,"rangeMax":2},{"rate":"0.00045700","amount":"0.03000217","rangeMin":2,"rangeMax":2},{"rate":"0.00045799","amount":"0.00454365","rangeMin":5,"rangeMax":5},{"rate":"0.00045800","amount":"0.31394478","rangeMin":2,"rangeMax":2},{"rate":"0.00046100","amount":"0.06284271","rangeMin":2,"rangeMax":2},{"rate":"0.00046149","amount":"0.10923269","rangeMin":2,"rangeMax":2},{"rate":"0.00046200","amount":"0.45535344","rangeMin":2,"rangeMax":60},{"rate":"0.00046300","amount":"0.23110000","rangeMin":2,"rangeMax":2},{"rate":"0.00046400","amount":"0.48000000","rangeMin":60,"rangeMax":60},{"rate":"0.00046600","amount":"0.20686534","rangeMin":2,"rangeMax":2},{"rate":"0.00046639","amount":"0.00544359","rangeMin":2,"rangeMax":2},{"rate":"0.00046700","amount":"0.03163966","rangeMin":2,"rangeMax":2},{"rate":"0.00046739","amount":"1.47336945","rangeMin":2,"rangeMax":2},{"rate":"0.00046775","amount":"0.01020135","rangeMin":2,"rangeMax":2},{"rate":"0.00046800","amount":"0.00721444","rangeMin":5,"rangeMax":5},{"rate":"0.00046900","amount":"0.00000002","rangeMin":2,"rangeMax":2},{"rate":"0.00047000","amount":"0.04278916","rangeMin":60,"rangeMax":60},{"rate":"0.00047500","amount":"0.01535784","rangeMin":2,"rangeMax":2},{"rate":"0.00047899","amount":"0.09894097","rangeMin":2,"rangeMax":2},{"rate":"0.00047999","amount":"1.25154781","rangeMin":2,"rangeMax":2},{"rate":"0.00048000","amount":"0.10491316","rangeMin":2,"rangeMax":2},{"rate":"0.00048099","amount":"1.37984373","rangeMin":2,"rangeMax":2},{"rate":"0.00048100","amount":"4.44520156","rangeMin":2,"rangeMax":2},{"rate":"0.00048198","amount":"0.01901905","rangeMin":2,"rangeMax":2},{"rate":"0.00048199","amount":"13.89942721","rangeMin":2,"rangeMax":2},{"rate":"0.00048200","amount":"3.42726287","rangeMin":2,"rangeMax":60},{"rate":"0.00048300","amount":"1.81679579","rangeMin":2,"rangeMax":32},{"rate":"0.00048400","amount":"4.79638300","rangeMin":2,"rangeMax":2},{"rate":"0.00048500","amount":"0.56860091","rangeMin":2,"rangeMax":2},{"rate":"0.00048600","amount":"0.50545263","rangeMin":2,"rangeMax":2}],"demands":[{"rate":"0.00014900","amount":"0.18976441","rangeMin":2,"rangeMax":2},{"rate":"0.00010300","amount":"0.13556749","rangeMin":2,"rangeMax":2},{"rate":"0.00010100","amount":"26.66019652","rangeMin":2,"rangeMax":2},{"rate":"0.00006000","amount":"49.99342005","rangeMin":2,"rangeMax":2}]}
{"offers":[{"rate":"0.00038499","amount":"0.00100000","rangeMin":2,"rangeMax":2},{"rate":"0.00038500","amount":"0.00202262","rangeMin":2,"rangeMax":2},{"rate":"0.00038600","amount":"0.97809855","rangeMin":2,"rangeMax":2},{"rate":"0.00038700","amount":"0.11781954","rangeMin":2,"rangeMax":2},{"rate":"0.00038800","amount":"0.00008255","rangeMin":2,"rangeMax":2},{"rate":"0.00038885","amount":"0.00383026","rangeMin":2,"rangeMax":2},{"rate":"0.00038900","amount":"0.00181875","rangeMin":2,"rangeMax":2},{"rate":"0.00039000","amount":"0.31843708","rangeMin":2,"rangeMax":3},{"rate":"0.00039200","amount":"0.03263231","rangeMin":2,"rangeMax":2},{"rate":"0.00039300","amount":"0.02604170","rangeMin":2,"rangeMax":2},{"rate":"0.00039400","amount":"0.06210503","rangeMin":2,"rangeMax":2},{"rate":"0.00039900","amount":"0.02034900","rangeMin":2,"rangeMax":2},{"rate":"0.00040000","amount":"0.14834252","rangeMin":2,"rangeMax":2},{"rate":"0.00040198","amount":"0.00174736","rangeMin":2,"rangeMax":2},{"rate":"0.00040600","amount":"0.01058767","rangeMin":2,"rangeMax":2},{"rate":"0.00040900","amount":"0.02221216","rangeMin":2,"rangeMax":2},{"rate":"0.00043175","amount":"0.01399529","rangeMin":2,"rangeMax":2},{"rate":"0.00044000","amount":"0.62522560","rangeMin":2,"rangeMax":2},{"rate":"0.00045100","amount":"0.03316436","rangeMin":2,"rangeMax":2},{"rate":"0.00045199","amount":"0.01366751","rangeMin":2,"rangeMax":2},{"rate":"0.00045200","amount":"3.40728067","rangeMin":2,"rangeMax":2},{"rate":"0.00045700","amount":"0.03000217","rangeMin":2,"rangeMax":2},{"rate":"0.00045799","amount":"0.00454365","rangeMin":5,"rangeMax":5},{"rate":"0.00045800","amount":"0.31394478","rangeMin":2,"rangeMax":2},{"rate":"0.00046100","amount":"0.06284271","rangeMin":2,"rangeMax":2},{"rate":"0.00046149","amount":"0.10923269","rangeMin":2,"rangeMax":2},{"rate":"0.00046200","amount":"0.45535344","rangeMin":2,"rangeMax":60},{"rate":"0.00046300","amount":"0.23110000","rangeMin":2,"rangeMax":2},{"rate":"0.00046400","amount":"0.48000000","rangeMin":60,"rangeMax":60},{"rate":"0.00046600","amount":"0.20686534","rangeMin":2,"rangeMax":2},{"rate":"0.00046639","amount":"0.00544359","rangeMin":2,"rangeMax":2},{"rate":"0.00046700","amount":"0.03163966","rangeMin":2,"rangeMax":2},{"rate":"0.00046739","amount":"1.47336945","rangeMin":2,"rangeMax":2},{"rate":"0.00046775","amount":"0.01020135","rangeMin":2,"rangeMax":2},{"rate":"0.00046800","amount":"0.00721444","rangeMin":5,"rangeMax":5},{"rate":"0.00046900","amount":"0.00000002","rangeMin":2,"rangeMax":2},{"rate":"0.00047000","amount":"0.04278916","rangeMin":60,"rangeMax":60},{"rate":"0.00047500","amount":"0.01535784","rangeMin":2,"rangeMax":2},{"rate":"0.00047899","amount":"0.09894097","rangeMin":2,"rangeMax":2},{"rate":"0.00047999","amount":"1.25154781","rangeMin":2,"rangeMax":2},{"rate":"0.00048000","amount":"0.10491316","rangeMin":2,"rangeMax":2},{"rate":"0.00048099","amount":"1.37984373","rangeMin":2,"rangeMax":2},{"rate":"0.00048100","amount":"4.44520156","rangeMin":2,"rangeMax":2},{"rate":"0.00048198","amount":"0.01901905","rangeMin":2,"rangeMax":2},{"rate":"0.00048199","amount":"13.91904729","rangeMin":2,"rangeMax":2},{"rate":"0.00048200","amount":"3.42726287","rangeMin":2,"rangeMax":60},{"rate":"0.00048300","amount":"1.81679579","rangeMin":2,"rangeMax":32},{"rate":"0.00048400","amount":"4.79638300","rangeMin":2,"rangeMax":2},{"rate":"0.00048500","amount":"0.56860091","rangeMin":2,"rangeMax":2},{"rate":"0.00048600","amount":"0.50545263","rangeMin":2,"rangeMax":2}],"demands":[{"rate":"0.00014900","amount":"0.18976441","rangeMin":2,"rangeMax":2},{"rate":"0.00010300","amount":"0.13556749","rangeMin":2,"rangeMax":2},{"rate":"0.00010100","amount":"26.66019652","rangeMin":2,"rangeMax":2},{"rate":"0.00006000","amount":"49.99342005","rangeMin":2,"rangeMax":2}]}


currency=BTC&command=returnLoanOrders


what is wrong with last 2 tries that do not work?
SEnergy
 
Posts: 12
Joined: Sun Jul 10, 2016 4:36 pm

Re: urlopen returns empty string

Postby wavic » Sun Aug 21, 2016 6:22 am

Hello!
I have edited my post. Look at printed encoded value at the end.
You are missing part of the address. The query part
Actually you POST this:

https://poloniex.com/publiccurrency=BTC ... LoanOrders

https://poloniex.com/public?currency=BTC&command=returnLoanOrders

So your urlopen() have to be
Code: Select all
print urllib2.urlopen("https://poloniex.com/public?", encoded).read()
wavic
 
Posts: 165
Joined: Wed May 25, 2016 8:51 pm

Re: urlopen returns empty string

Postby SEnergy » Sun Aug 21, 2016 9:01 am

i'm not sure this is the cause, as the same code works with other APIs, however I tried to change it as you mentioned with the same result...

Code: Select all
#this does not
print urllib2.urlopen(urllib2.Request("https://poloniex.com/public?", encoded)).read()
print urllib2.urlopen("https://poloniex.com/public?", encoded).read()


still gives empty strings
SEnergy
 
Posts: 12
Joined: Sun Jul 10, 2016 4:36 pm

Re: urlopen returns empty string

Postby wavic » Sun Aug 21, 2016 9:15 am

What if you pack whole url and pass it to urlopen()?
Code: Select all
urlopen(join(base_url, encoded))
wavic
 
Posts: 165
Joined: Wed May 25, 2016 8:51 pm

Re: urlopen returns empty string

Postby SEnergy » Sun Aug 21, 2016 9:27 am

Code: Select all
urllib2.urlopen("".join(("https://poloniex.com/public?", encoded))).read()
works, however I'm more interested in why the code as it was didn't work, when it works just fine with this
Code: Select all
params = {"method":"stats.global.current", "location":0}
encoded = urllib.urlencode(params)
print urllib2.urlopen(urllib2.Request("https://www.nicehash.com/api", encoded)).read()

for example...
SEnergy
 
Posts: 12
Joined: Sun Jul 10, 2016 4:36 pm

Re: urlopen returns empty string

Postby wavic » Sun Aug 21, 2016 12:15 pm

https://poloniex.com/public?currency=BT ... LoanOrders returns a bunch of json. I opened in the browser

There is a example at https://www.nicehash.com/?p=api :

https://www.nicehash.com/api?method=met ... eter2value

I still don't see the question mark in your code. Add it to the base url and try again. This one:

https://www.nicehash.com/api?

Code: Select all
~ ❯❯❯ ipython
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
Type "copyright", "credits" or "license" for more information.

IPython 4.2.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import urllib

In [2]: params = {"method":"stats.global.current", "location":0}

In [3]: encoded = urllib.urlencode(params)

In [4]: print urllib.urlopen("https://www.nicehash.com/api?", encoded)).read()  File "<ipython-input-4-15692e5afb66>", line 1
    print urllib.urlopen("https://www.nicehash.com/api?", encoded)).read()
                                                                  ^
SyntaxError: invalid syntax


In [5]: print urllib.urlopen("https://www.nicehash.com/api?", encoded).read()
{"result":{"stats":[{"profitability_above_ltc":"9.91","price":"0.0676","profitability_ltc":"0.0615","algo":0,"speed":"47.66579962"},{"price":"1.2191","profitability_btc":"1.1568","profitability_above_btc":"5.38","algo":1,"speed":"3385034.00000000"},{"profitability_above_ltc":"-100.00","price":"0.0000","profitability_ltc":"0.0615","algo":2,"speed":"0.00000000"},{"profitability_above_ltc":"626.02","price":"0.0298","profitability_ltc":"0.0615","algo":3,"speed":"74.04396200"},{"profitability_above_ltc":"237.45","price":"0.0319","profitability_ltc":"0.0615","algo":4,"speed":"0.33612100"},{"profitability_above_ltc":"1371.23","price":"0.0018","profitability_ltc":"0.0615","algo":5,"speed":"13.62220700"},{"profitability_above_ltc":"4.86","price":"0.0129","profitability_ltc":"0.0615","algo":6,"speed":"0.00309400"},{"profitability_above_ltc":"447.64","price":"0.0241","profitability_ltc":"0.0615","algo":7,"speed":"0.19124700"},{"profitability_above_ltc":"2488.02","price":"3.5377","profitability_ltc":"0.0615","algo":8,"speed":"0.23802105"},{"profitability_above_ltc":"978.69","price":"0.3792","profitability_ltc":"0.0615","algo":9,"speed":"2.00419460"},{"profitability_above_ltc":"-100.00","price":"0.0000","profitability_ltc":"0.0615","algo":10,"speed":"0.00000000"},{"profitability_above_ltc":"928.23","price":"0.0395","profitability_ltc":"0.0615","algo":11,"speed":"0.24506300"},{"profitability_above_ltc":"935.30","price":"0.0303","profitability_ltc":"0.0615","algo":12,"speed":"1.79530801"},{"profitability_above_ltc":"-100.00","price":"0.0000","profitability_ltc":"0.0615","algo":13,"speed":"0.00000000"},{"profitability_above_ltc":"2433.51","price":"0.0917","profitability_ltc":"0.0615","algo":14,"speed":"14.31462700"},{"profitability_above_ltc":"-100.00","price":"0.0000","profitability_ltc":"0.0615","algo":15,"speed":"0.00000000"},{"profitability_above_ltc":"1037.97","price":"0.0002","profitability_ltc":"0.0615","algo":16,"speed":"0.28900000"},{"profitability_above_ltc":"826.63","price":"0.0003","profitability_ltc":"0.0615","algo":17,"speed":"0.00000000"},{"profitability_above_ltc":"871665.03","price":"0.1532","profitability_ltc":"0.0615","algo":18,"speed":"12.78900000"},{"profitability_above_ltc":"-100.00","price":"0.0006","profitability_ltc":"0.0615","algo":19,"speed":"0.00000393"},{"price":"0.1209","algo":20,"speed":"53.54274400","profitability_eth":"0.1322","profitability_above_eth":"-8.59"},{"profitability_above_ltc":"2947368.51","price":"0.9543","profitability_ltc":"0.0615","algo":21,"speed":"177.62104788"},{"price":"0.0000","algo":999,"speed":"0.00000000"}]},"method":"stats.global.current"}

In [6]:



It works.
wavic
 
Posts: 165
Joined: Wed May 25, 2016 8:51 pm

Re: urlopen returns empty string

Postby SEnergy » Sun Aug 21, 2016 12:39 pm

wavic wrote:https://poloniex.com/public?currency=BTC&command=returnLoanOrders returns a bunch of json. I opened in the browser

There is a example at https://www.nicehash.com/?p=api :

https://www.nicehash.com/api?method=met ... eter2value

I still don't see the question mark in your code. Add it to the base url and try again. This one:

https://www.nicehash.com/api?


It works.


I said that the nicehash one IS working, which is weird, because the same code using poloniex api is not, so the question mark isn't required, as the urlopen puts that between url and parameters by default, there's a python wrapper for poloniex api http://pastebin.com/fbkheaRb that works on the same principle
look at the api_query function, it also does not have the question mark, yet it is working flawlessly

to make this clear:

Code: Select all
>>> import urllib
>>> import urllib2
>>> params = {"command":"returnTicker"}
>>> encoded = urllib.urlencode(params)
>>> print urllib2.urlopen(urllib2.Request("https://poloniex.com/public", encoded)).read()

>>> params = {"method":"stats.global.current", "location":0}
>>> encoded = urllib.urlencode(params)
>>> urllib2.urlopen(urllib2.Request("https://www.nicehash.com/api", encoded)).read()
'{"result":{"stats":[{"profitability_above_ltc":"8.47","price":"0.0667","profitability_ltc":"0.0615","algo":0,"speed":"47.08401400"},{"price":"1.2170","profitability_btc":"1.1568","profitability_above_btc":"5.21","algo":1,"speed":"3364203.00000000"},{"profitability_above_ltc":"-100.00","price":"0.0000","profitability_ltc":"0.0615","algo":2,"speed":"0.00000000"},{"profitability_above_ltc":"615.09","price":"0.0293","profitability_ltc":"0.0615","algo":3,"speed":"71.05113700"},{"profitability_above_ltc":"237.43","price":"0.0319","profitability_ltc":"0.0615","algo":4,"speed":"0.34851900"},{"profitability_above_ltc":"1371.23","price":"0.0018","profitability_ltc":"0.0615","algo":5,"speed":"13.02190700"},{"profitability_above_ltc":"22.71","price":"0.0151","profitability_ltc":"0.0615","algo":6,"speed":"0.00285000"},{"profitability_above_ltc":"472.15","price":"0.0251","profitability_ltc":"0.0615","algo":7,"speed":"0.22317400"},{"profitability_above_ltc":"2518.24","price":"3.5790","profitability_ltc":"0.0615","algo":8,"speed":"0.30346900"},{"profitability_above_ltc":"980.43","price":"0.3798","profitability_ltc":"0.0615","algo":9,"speed":"1.93243200"},{"profitability_above_ltc":"-100.00","price":"0.0000","profitability_ltc":"0.0615","algo":10,"speed":"0.00000000"},{"profitability_above_ltc":"901.02","price":"0.0385","profitability_ltc":"0.0615","algo":11,"speed":"0.17546000"},{"profitability_above_ltc":"901.37","price":"0.0293","profitability_ltc":"0.0615","algo":12,"speed":"1.63658500"},{"profitability_above_ltc":"-100.00","price":"0.0000","profitability_ltc":"0.0615","algo":13,"speed":"0.00000000"},{"profitability_above_ltc":"2526.40","price":"0.0950","profitability_ltc":"0.0615","algo":14,"speed":"13.19847076"},{"profitability_above_ltc":"-100.00","price":"0.0000","profitability_ltc":"0.0615","algo":15,"speed":"0.00000000"},{"profitability_above_ltc":"1663.85","price":"0.0003","profitability_ltc":"0.0615","algo":16,"speed":"2.74900000"},{"profitability_above_ltc":"857.52","price":"0.0003","profitability_ltc":"0.0615","algo":17,"speed":"0.57700000"},{"profitability_above_ltc":"799085.28","price":"0.1405","profitability_ltc":"0.0615","algo":18,"speed":"17.68424077"},{"profitability_above_ltc":"-100.00","price":"0.0006","profitability_ltc":"0.0615","algo":19,"speed":"0.00000384"},{"price":"0.1212","algo":20,"speed":"52.17901100","profitability_eth":"0.1322","profitability_above_eth":"-8.36"},{"profitability_above_ltc":"2947414.84","price":"0.9543","profitability_ltc":"0.0615","algo":21,"speed":"180.71404932"},{"price":"0.0000","algo":999,"speed":"0.00000000"}]},"method":"stats.global.current"}'
>>>


the ONLY difference is the API and parameters, the code to access it is the SAME, both should work, so what's wrong with the poloniex one?
SEnergy
 
Posts: 12
Joined: Sun Jul 10, 2016 4:36 pm

Re: urlopen returns empty string

Postby wavic » Sun Aug 21, 2016 3:14 pm

I don't want to argue with you but there is '?' in the api_query() method. Look carefully. Just between '/public' and 'command='

Code: Select all
def api_query(self, command, req={}):
 
        if(command == "returnTicker" or command == "return24Volume"):
            ret = urllib2.urlopen(urllib2.Request('https://poloniex.com/public?command=' + command))
            return json.loads(ret.read())
        elif(command == "returnOrderBook"):
            ret = urllib2.urlopen(urllib2.Request('https://poloniex.com/public?command=' + command + '&currencyPair=' +
wavic
 
Posts: 165
Joined: Wed May 25, 2016 8:51 pm

Re: urlopen returns empty string

Postby SEnergy » Sun Aug 21, 2016 8:43 pm

sure there is, for those 3 specific commands, as those are public commands, in the else (where the secret is hashed into the header), where you basically do
Code: Select all
poloniex.api_query(command, encoded)

is no question mark, the Request kinda takes the url and params and makes it one, including the question mark, otherwise it wouldn't work now, would it?

now again, look at the code I posted, the code is basically the SAME thing, the only difference between them is the API used, so if the question mark is not included by the urlopen/request, then neither the second one (nicehash) would work, but it does, doesn't it?

just for the lulz let's try it without any API, let's try to get this thread ...

Code: Select all
>>> import urllib
>>> import urllib2
>>> params = {"f":"17", "t":"20720"}
>>> encoded = urllib.urlencode(params)
>>> print urllib2.urlopen(urllib2.Request("http://www.python-forum.org/viewtopic.php", encoded)).read()


as you can guess, it returns the HTML page of this thread just fine, without the question mark, so the question mark should not be a problem (again, look in the ELSE statement of api_query, that's what I'm trying to do)
SEnergy
 
Posts: 12
Joined: Sun Jul 10, 2016 4:36 pm


Return to Networking

Who is online

Users browsing this forum: No registered users and 2 guests

cron