Initial testnet support.

[inertia186]

This is my initial stab at adding testnet support to hivemind. In order to try it out, you have to run a stable testnet. Unfortunately, at the moment, the official testnet responds with Unable to acquire database lock.

So, I am testing with tintoy instead, which allows me to launch a tiny testnet with ~2,000 accounts. To run:

docker run -d -p 8091:8091 inertia/tintoy:latest
# wait for it to load after < 10 minutes or so
hive sync --steemd_url http://localhost:8091

At this point, the problem appears to be some kind of "timeout" situation when trying to connect to localhost:8091. I believe it's expecting SSL, but I don't see where it's getting that.

I'm using ephemeral ports in this example:

bucky:hivemind (testnet-support)$ hive sync --steemd-url http://localhost:32768
INFO:root:loaded configuration:
Command Line Args:   sync --steemd-url http://localhost:32768
Environment Variables:
  DATABASE_URL:      postgresql://localhost:5432/hive
Defaults:
  mode:              ['sync']
  --http-server-port:8080
  --max-workers:     4
  --max-batch:       50
  --trail-blocks:    2
  --sync-to-s3:      False
  --log-level:       INFO
  --test-disable-sync:False

INFO:hive.steem.http_client:using node: http://localhost:32768
INFO:hive.db.db_state:[INIT] Welcome to hive!
INFO:hive.steem.http_client:using node: http://localhost:32768
ERROR:hive.steem.http_client:get_config[1] failed in 30.0s. try 1. {'secs': 30.008, 'try': 1} - ReadTimeoutError("HTTPConnectionPool(host='localhost', port=32768): Read timed out. (read timeout=30)")
^C

Any suggestions?

[roadscape]

This looks great. I do have some suggested changes but there are semi-major updates incoming shortly (this week). Will circle back ASAP.

[inertia186]

Yep, makes sense. Thanks for taking a peek.

[inertia186]

I think maybe my previous problem had to do with ephemeral ports. When I launch docker with -p 8091:8091, I'm able to sync.

I think the remaining issue I'm seeing is because my testnet has no price feeds. At initial sync, I see:

DETAIL:  A field with precision 8, scale 6 must round to an absolute value less than 10^2.

I need to make sure price feeds are working in tintoy. In the meantime, I patched with this and initial sync works:

diff --git a/hive/indexer/sync.py b/hive/indexer/sync.py
index 0ff873e..776e025 100644
--- a/hive/indexer/sync.py
+++ b/hive/indexer/sync.py
@@ -212,12 +212,18 @@ class Sync:
                            sps=state['sbd_per_steem'],
                            dgpo=json.dumps(state['dgpo']))
         elif chain == 'testnet':
-            self._db.query("""UPDATE hive_state SET block_num = :block_num,
-                           tests_per_mvest = :tpm, usd_per_steem = :ups,
-                           tbd_per_steem = :tps, dgpo = :dgpo""",
-                           block_num=state['dgpo']['head_block_number'],
-                           tpm=state['tests_per_mvest'],
-                           ups=state['usd_per_steem'],
-                           tps=state['tbd_per_steem'],
-                           dgpo=json.dumps(state['dgpo']))
+            if state['usd_per_steem'] == '0.000' or state['tbd_per_steem'] == '0.000':
+                self._db.query("""UPDATE hive_state SET block_num = :block_num,
+                               dgpo = :dgpo""",
+                               block_num=state['dgpo']['head_block_number'],
+                               dgpo=json.dumps(state['dgpo']))
+            else:
+                self._db.query("""UPDATE hive_state SET block_num = :block_num,
+                               tests_per_mvest = :tpm, usd_per_steem = :ups,
+                               tbd_per_steem = :tps, dgpo = :dgpo""",
+                               block_num=state['dgpo']['head_block_number'],
+                               tpm=state['tests_per_mvest'],
+                               ups=state['usd_per_steem'],
+                               tps=state['tbd_per_steem'],
+                               dgpo=json.dumps(state['dgpo']))
         return state['dgpo']['head_block_number']

[roadscape]

Let's try avoid testnet branching as much as possible. Testnet support should be achievable with very minimal LOC added. For instance with steem vs tests columns: let's just leave names as-is. In the future when there's a schema change we could consider renaming them to generics (as you started to w/ methods), e.g.:

• steem/tests -> liquid
• sbd/tbd -> debt