Initial commit

This commit is contained in:
Tim Glaser
2020-02-09 21:11:22 -08:00
commit 8eb6c5e9a2
23 changed files with 1687 additions and 0 deletions
+14
View File
@@ -0,0 +1,14 @@
**sublime**
*.pyc
dist
*.egg-info
dist
MANIFEST
build
.eggs
.coverage
.vscode/
env
flake8.out
pylint.out
+20
View File
@@ -0,0 +1,20 @@
Copyright (c) 2020 PostHog (part of Hiberly Inc)
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+1
View File
@@ -0,0 +1 @@
include README.md
+13
View File
@@ -0,0 +1,13 @@
test:
pylint --rcfile=.pylintrc --reports=y --exit-zero analytics | tee pylint.out
flake8 --max-complexity=10 --statistics analytics > flake8.out || true
coverage run --branch --include=analytics/\* --omit=*/test* setup.py test
release:
python setup.py sdist bdist_wheel
twine upload dist/*
e2e_test:
.buildscripts/e2e.sh
.PHONY: test release e2e_test
+84
View File
@@ -0,0 +1,84 @@
# PostHog Python
Official PostHog Python library to capture and send events to any PostHhog instance (including PostHog.com).
This library uses an internal queue to make calls non-blocking and fast. It also batches requests and flushes asynchronously, making it perfect to use in any part of your web app or other server side application that needs performance.
## Installation
```bash
pip install posthog
```
In your app, import the posthog library and set your api key **before** making any calls.
```python
import posthog
posthog.write_key = 'YOUR API KEY'
```
You can find your key in the /setup page in PostHog.
To debug, you can set debug mode.
```python
posthog.debug = True
```
## Making calls
### Capture
Capture allows you to capture anything a user does within your system, which you can later use in PostHog to find patterns in usage, work out which features to improve or where people are giving up.
A `capture` call requires
- `distinct id` which uniquely identifies your user
- `event name` to make sure
- We recommend using [verb] [noun], like `movie played` or `movie updated` to easily identify what your events mean later on.
Optionally you can submit
- `properties`, which can be a dict with any information you'd like to add
For example:
```python
posthog.capture('distinct id', 'movie played', {'movie_id': '123', 'category': 'romcom'})
```
### Identify
Identify lets you add metadata on your users so you can more easily identify who they are in PostHog, and even do things like segment users by these properties.
An `identify` call requires
- `distinct id` which uniquely identifies your user
- `properties` with a dict with any key: value pairs
For example:
```python
posthog.capture('distinct id', {
'email': 'dwayne@gmail.com',
'name': 'Dwayne Johnson'
})
```
The most obvious place to make this call is whenever a user signs up, or when they update their information.
### Alias
To marry up whatever a user does before they sign up or log in with what they do after you need to make an alias call. This will allow you to answer questions like "Which marketing channels leads to users churning after a month?" or "What do users do on our website before signing up?"
In a purely back-end implementation, this means whenever an anonymous user does something, you'll want to send a session ID ([Django](https://stackoverflow.com/questions/526179/in-django-how-can-i-find-out-the-request-session-sessionid-and-use-it-as-a-vari), [Flask](https://stackoverflow.com/questions/15156132/flask-login-how-to-get-session-id)) with the capture call. Then, when that users signs up, you want to do an alias call with the session ID and the newly created user ID.
The same concept applies for when a user logs in.
An `alias` call requires
- `previous distinct id` the unique ID of the user before
- `distinct id` the current unique id
For example:
```python
posthog.alias('anonymous session id', 'distinct id')
```
## Thank you
This library is largely based on the `analytics-python` package.
+8
View File
@@ -0,0 +1,8 @@
Releasing
=========
1. Update `VERSION` in `posthog/version.py` to the new version.
2. Update the `HISTORY.md` for the impending release.
3. `git commit -am "Release X.Y.Z."` (where X.Y.Z is the new version)
4. `git tag -a X.Y.Z -m "Version X.Y.Z"` (where X.Y.Z is the new version).
5. `make release`.
Executable
+5
View File
@@ -0,0 +1,5 @@
#!/bin/sh
set -e
python ./simulator.py "$@"
+20
View File
@@ -0,0 +1,20 @@
# PostHog Python library example
# Import the library
import posthog
# You can find this key on the /setup page in PostHog
posthog.write_key = '<your key>'
# Where you host PostHog, with no trailing /.
# You can remove this line if you're using posthog.com
posthog.host = 'http://127.0.0.1:8000'
# Capture an event
posthog.capture('distinct_id', 'event', {'property1': 'value', 'property2': 'value'})
# Alias a previous distinct id with a new one
posthog.alias('distinct_id', 'new_distinct_id')
# Add properties to the person
posthog.identify('distinct_id', {'email': 'something@something.com'})
+72
View File
@@ -0,0 +1,72 @@
from posthog.version import VERSION
from posthog.client import Client
__version__ = VERSION
"""Settings."""
write_key = None
host = None
on_error = None
debug = False
send = True
sync_mode = False
default_client = None
def capture(*args, **kwargs):
"""Send a capture call."""
_proxy('capture', *args, **kwargs)
def identify(*args, **kwargs):
"""Send a identify call."""
_proxy('identify', *args, **kwargs)
def group(*args, **kwargs):
"""Send a group call."""
_proxy('group', *args, **kwargs)
def alias(*args, **kwargs):
"""Send a alias call."""
_proxy('alias', *args, **kwargs)
def page(*args, **kwargs):
"""Send a page call."""
_proxy('page', *args, **kwargs)
def screen(*args, **kwargs):
"""Send a screen call."""
_proxy('screen', *args, **kwargs)
def flush():
"""Tell the client to flush."""
_proxy('flush')
def join():
"""Block program until the client clears the queue"""
_proxy('join')
def shutdown():
"""Flush all messages and cleanly shutdown the client"""
_proxy('flush')
_proxy('join')
def _proxy(method, *args, **kwargs):
"""Create an analytics client if one doesn't exist and send to it."""
global default_client
if not default_client:
default_client = Client(write_key, host=host, debug=debug,
on_error=on_error, send=send,
sync_mode=sync_mode)
fn = getattr(default_client, method)
fn(*args, **kwargs)
+287
View File
@@ -0,0 +1,287 @@
from datetime import datetime
from uuid import uuid4
import logging
import numbers
import atexit
from dateutil.tz import tzutc
from six import string_types
from posthog.utils import guess_timezone, clean
from posthog.consumer import Consumer
from posthog.request import post
from posthog.version import VERSION
try:
import queue
except ImportError:
import Queue as queue
ID_TYPES = (numbers.Number, string_types)
class Client(object):
"""Create a new PostHog client."""
log = logging.getLogger('posthog')
def __init__(self, write_key=None, host=None, debug=False,
max_queue_size=10000, send=True, on_error=None, flush_at=100,
flush_interval=0.5, gzip=False, max_retries=3,
sync_mode=False, timeout=15, thread=1):
require('write_key', write_key, string_types)
self.queue = queue.Queue(max_queue_size)
self.write_key = write_key
self.on_error = on_error
self.debug = debug
self.send = send
self.sync_mode = sync_mode
self.host = host
self.gzip = gzip
self.timeout = timeout
if debug:
self.log.setLevel(logging.DEBUG)
if sync_mode:
self.consumers = None
else:
# On program exit, allow the consumer thread to exit cleanly.
# This prevents exceptions and a messy shutdown when the
# interpreter is destroyed before the daemon thread finishes
# execution. However, it is *not* the same as flushing the queue!
# To guarantee all messages have been delivered, you'll still need
# to call flush().
if send:
atexit.register(self.join)
for n in range(thread):
self.consumers = []
consumer = Consumer(
self.queue, write_key, host=host, on_error=on_error,
flush_at=flush_at, flush_interval=flush_interval,
gzip=gzip, retries=max_retries, timeout=timeout,
)
self.consumers.append(consumer)
# if we've disabled sending, just don't start the consumer
if send:
consumer.start()
def identify(self, distinct_id=None, traits=None, context=None, timestamp=None,
message_id=None):
traits = traits or {}
context = context or {}
require('distinct_id', distinct_id, ID_TYPES)
require('traits', traits, dict)
msg = {
'timestamp': timestamp,
'context': context,
'type': 'identify',
'distinct_id': distinct_id,
'$set': traits,
'event': '$identify',
'messageId': message_id,
}
return self._enqueue(msg)
def capture(self, distinct_id=None, event=None, properties=None, context=None,
timestamp=None, message_id=None):
properties = properties or {}
context = context or {}
require('distinct_id', distinct_id, ID_TYPES)
require('properties', properties, dict)
require('event', event, string_types)
msg = {
'properties': properties,
'timestamp': timestamp,
'context': context,
'distinct_id': distinct_id,
'type': 'capture',
'event': event,
'messageId': message_id,
}
return self._enqueue(msg)
def alias(self, previous_id=None, distinct_id=None, context=None,
timestamp=None, message_id=None):
context = context or {}
require('previous_id', previous_id, ID_TYPES)
require('distinct_id', distinct_id, ID_TYPES)
msg = {
'properties': {
'distinct_id': previous_id,
'alias': distinct_id,
},
'timestamp': timestamp,
'context': context,
'type': 'alias',
'event': '$create_alias'
}
return self._enqueue(msg)
def group(self, distinct_id=None, group_id=None, traits=None, context=None,
timestamp=None, message_id=None):
traits = traits or {}
context = context or {}
require('distinct_id', distinct_id, ID_TYPES)
require('group_id', group_id, ID_TYPES)
require('traits', traits, dict)
msg = {
'timestamp': timestamp,
'groupId': group_id,
'context': context,
'distinct_id': distinct_id,
'traits': traits,
'type': 'group',
'messageId': message_id,
}
return self._enqueue(msg)
def page(self, distinct_id=None, category=None, name=None, properties=None,
context=None, timestamp=None, message_id=None):
properties = properties or {}
context = context or {}
require('distinct_id', distinct_id, ID_TYPES)
require('properties', properties, dict)
if name:
require('name', name, string_types)
if category:
require('category', category, string_types)
msg = {
'properties': properties,
'timestamp': timestamp,
'category': category,
'context': context,
'distinct_id': distinct_id,
'type': 'page',
'name': name,
'messageId': message_id,
}
return self._enqueue(msg)
def screen(self, distinct_id=None, category=None, name=None, properties=None,
context=None, timestamp=None, message_id=None):
properties = properties or {}
context = context or {}
require('distinct_id', distinct_id, ID_TYPES)
require('properties', properties, dict)
if name:
require('name', name, string_types)
if category:
require('category', category, string_types)
msg = {
'properties': properties,
'timestamp': timestamp,
'category': category,
'context': context,
'distinct_id': distinct_id,
'type': 'screen',
'name': name,
'messageId': message_id,
}
return self._enqueue(msg)
def _enqueue(self, msg):
"""Push a new `msg` onto the queue, return `(success, msg)`"""
timestamp = msg['timestamp']
if timestamp is None:
timestamp = datetime.utcnow().replace(tzinfo=tzutc())
message_id = msg.get('messageId')
if message_id is None:
message_id = uuid4()
require('type', msg['type'], string_types)
require('timestamp', timestamp, datetime)
require('context', msg['context'], dict)
# add common
timestamp = guess_timezone(timestamp)
msg['timestamp'] = timestamp.isoformat()
msg['messageId'] = stringify_id(message_id)
msg['library'] = 'posthog-python'
msg['library_version'] = VERSION
msg['api_key'] = self.write_key
msg['distinct_id'] = stringify_id(msg.get('distinct_id', None))
msg = clean(msg)
self.log.debug('queueing: %s', msg)
# if send is False, return msg as if it was successfully queued
if not self.send:
return True, msg
if self.sync_mode:
self.log.debug('enqueued with blocking %s.', msg['type'])
post(self.host, gzip=self.gzip,
timeout=self.timeout, batch=[msg])
return True, msg
try:
self.queue.put(msg, block=False)
self.log.debug('enqueued %s.', msg['type'])
return True, msg
except queue.Full:
self.log.warning('analytics-python queue is full')
return False, msg
def flush(self):
"""Forces a flush from the internal queue to the server"""
queue = self.queue
size = queue.qsize()
queue.join()
# Note that this message may not be precise, because of threading.
self.log.debug('successfully flushed about %s items.', size)
def join(self):
"""Ends the consumer thread once the queue is empty.
Blocks execution until finished
"""
for consumer in self.consumers:
consumer.pause()
try:
consumer.join()
except RuntimeError:
# consumer thread has not started
pass
def shutdown(self):
"""Flush all messages and cleanly shutdown the client"""
self.flush()
self.join()
def require(name, field, data_type):
"""Require that the named `field` has the right `data_type`"""
if not isinstance(field, data_type):
msg = '{0} must have {1}, got: {2}'.format(name, data_type, field)
raise AssertionError(msg)
def stringify_id(val):
if val is None:
return None
if isinstance(val, string_types):
return val
return str(val)
+134
View File
@@ -0,0 +1,134 @@
import logging
from threading import Thread
import monotonic
import backoff
import json
from posthog.request import post, APIError, DatetimeSerializer
try:
from queue import Empty
except ImportError:
from Queue import Empty
MAX_MSG_SIZE = 32 << 10
# Our servers only accept batches less than 500KB. Here limit is set slightly
# lower to leave space for extra data that will be added later, eg. "sentAt".
BATCH_SIZE_LIMIT = 475000
class Consumer(Thread):
"""Consumes the messages from the client's queue."""
log = logging.getLogger('posthog')
def __init__(self, queue, write_key, flush_at=100, host=None,
on_error=None, flush_interval=0.5, gzip=False, retries=10,
timeout=15):
"""Create a consumer thread."""
Thread.__init__(self)
# Make consumer a daemon thread so that it doesn't block program exit
self.daemon = True
self.flush_at = flush_at
self.flush_interval = flush_interval
self.write_key = write_key
self.host = host
self.on_error = on_error
self.queue = queue
self.gzip = gzip
# It's important to set running in the constructor: if we are asked to
# pause immediately after construction, we might set running to True in
# run() *after* we set it to False in pause... and keep running
# forever.
self.running = True
self.retries = retries
self.timeout = timeout
def run(self):
"""Runs the consumer."""
self.log.debug('consumer is running...')
while self.running:
self.upload()
self.log.debug('consumer exited.')
def pause(self):
"""Pause the consumer."""
self.running = False
def upload(self):
"""Upload the next batch of items, return whether successful."""
success = False
batch = self.next()
if len(batch) == 0:
return False
try:
self.request(batch)
success = True
except Exception as e:
self.log.error('error uploading: %s', e)
success = False
if self.on_error:
self.on_error(e, batch)
finally:
# mark items as acknowledged from queue
for item in batch:
self.queue.task_done()
return success
def next(self):
"""Return the next batch of items to upload."""
queue = self.queue
items = []
start_time = monotonic.monotonic()
total_size = 0
while len(items) < self.flush_at:
elapsed = monotonic.monotonic() - start_time
if elapsed >= self.flush_interval:
break
try:
item = queue.get(
block=True, timeout=self.flush_interval - elapsed)
item_size = len(json.dumps(
item, cls=DatetimeSerializer).encode())
if item_size > MAX_MSG_SIZE:
self.log.error(
'Item exceeds 32kb limit, dropping. (%s)', str(item))
continue
items.append(item)
total_size += item_size
if total_size >= BATCH_SIZE_LIMIT:
self.log.debug(
'hit batch size limit (size: %d)', total_size)
break
except Empty:
break
return items
def request(self, batch):
"""Attempt to upload the batch and retry before raising an error """
def fatal_exception(exc):
if isinstance(exc, APIError):
# retry on server errors and client errors
# with 429 status code (rate limited),
# don't retry on other client errors
return (400 <= exc.status < 500) and exc.status != 429
else:
# retry on all other errors (eg. network)
return False
@backoff.on_exception(
backoff.expo,
Exception,
max_tries=self.retries + 1,
giveup=fatal_exception)
def send_request():
post(self.host, gzip=self.gzip,
timeout=self.timeout, batch=batch)
send_request()
+69
View File
@@ -0,0 +1,69 @@
from datetime import date, datetime
from dateutil.tz import tzutc
import logging
import json
from gzip import GzipFile
from requests.auth import HTTPBasicAuth
from requests import sessions
from io import BytesIO
from posthog.version import VERSION
from posthog.utils import remove_trailing_slash
_session = sessions.Session()
def post(host=None, gzip=False, timeout=15, **kwargs):
"""Post the `kwargs` to the API"""
log = logging.getLogger('posthog')
body = kwargs
body["sentAt"] = datetime.utcnow().replace(tzinfo=tzutc()).isoformat()
url = remove_trailing_slash(host or 'https://t.posthog.com') + '/batch/'
data = json.dumps(body, cls=DatetimeSerializer)
log.debug('making request: %s', data)
headers = {
'Content-Type': 'application/json',
'User-Agent': 'analytics-python/' + VERSION
}
if gzip:
headers['Content-Encoding'] = 'gzip'
buf = BytesIO()
with GzipFile(fileobj=buf, mode='w') as gz:
# 'data' was produced by json.dumps(),
# whose default encoding is utf-8.
gz.write(data.encode('utf-8'))
data = buf.getvalue()
res = _session.post(url, data=data,
headers=headers, timeout=timeout)
if res.status_code == 200:
log.debug('data uploaded successfully')
return res
try:
payload = res.json()
log.debug('received response: %s', payload)
raise APIError(res.status_code, payload['code'], payload['message'])
except ValueError:
raise APIError(res.status_code, 'unknown', res.text)
class APIError(Exception):
def __init__(self, status, code, message):
self.message = message
self.status = status
self.code = code
def __str__(self):
msg = "[PostHog] {0}: {1} ({2})"
return msg.format(self.code, self.message, self.status)
class DatetimeSerializer(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, (date, datetime)):
return obj.isoformat()
return json.JSONEncoder.default(self, obj)
+14
View File
@@ -0,0 +1,14 @@
import unittest
import pkgutil
import logging
import sys
def all_names():
for _, modname, _ in pkgutil.iter_modules(__path__):
yield 'analytics.test.' + modname
def all():
logging.basicConfig(stream=sys.stderr)
return unittest.defaultTestLoader.loadTestsFromNames(all_names())
+316
View File
@@ -0,0 +1,316 @@
from datetime import date, datetime
import unittest
import six
import mock
import time
from posthog.version import VERSION
from posthoglient import Client
class TestClient(unittest.TestCase):
def fail(self, e, batch):
"""Mark the failure handler"""
self.failed = True
def setUp(self):
self.failed = False
self.client = Client('testsecret', on_error=self.fail)
def test_requires_write_key(self):
self.assertRaises(AssertionError, Client)
def test_empty_flush(self):
self.client.flush()
def test_basic_track(self):
client = self.client
success, msg = client.track('distinct_id', 'python test event')
client.flush()
self.assertTrue(success)
self.assertFalse(self.failed)
self.assertEqual(msg['event'], 'python test event')
self.assertTrue(isinstance(msg['timestamp'], str))
self.assertTrue(isinstance(msg['messageId'], str))
self.assertEqual(msg['distinct_id'], 'distinct_id')
self.assertEqual(msg['properties'], {})
self.assertEqual(msg['type'], 'track')
def test_stringifies_distinct_id(self):
# A large number that loses precision in node:
# node -e "console.log(157963456373623802 + 1)" > 157963456373623800
client = self.client
success, msg = client.track(
distinct_id=157963456373623802, event='python test event')
client.flush()
self.assertTrue(success)
self.assertFalse(self.failed)
self.assertEqual(msg['distinct_id'], '157963456373623802')
def test_advanced_track(self):
client = self.client
success, msg = client.track(
'distinct_id', 'python test event', {'property': 'value'},
{'ip': '192.168.0.1'}, datetime(2014, 9, 3),
'messageId')
self.assertTrue(success)
self.assertEqual(msg['timestamp'], '2014-09-03T00:00:00+00:00')
self.assertEqual(msg['properties'], {'property': 'value'})
self.assertEqual(msg['context']['ip'], '192.168.0.1')
self.assertEqual(msg['event'], 'python test event')
self.assertEqual(msg['context']['library'], {
'name': 'analytics-python',
'version': VERSION
})
self.assertEqual(msg['messageId'], 'messageId')
self.assertEqual(msg['distinct_id'], 'distinct_id')
self.assertEqual(msg['type'], 'track')
def test_basic_identify(self):
client = self.client
success, msg = client.identify('distinct_id', {'trait': 'value'})
client.flush()
self.assertTrue(success)
self.assertFalse(self.failed)
self.assertEqual(msg['traits'], {'trait': 'value'})
self.assertTrue(isinstance(msg['timestamp'], str))
self.assertTrue(isinstance(msg['messageId'], str))
self.assertEqual(msg['distinct_id'], 'distinct_id')
self.assertEqual(msg['type'], 'identify')
def test_advanced_identify(self):
client = self.client
success, msg = client.identify(
'distinct_id', {'trait': 'value'}, {'ip': '192.168.0.1'},
datetime(2014, 9, 3), 'messageId')
self.assertTrue(success)
self.assertEqual(msg['timestamp'], '2014-09-03T00:00:00+00:00')
self.assertEqual(msg['context']['ip'], '192.168.0.1')
self.assertEqual(msg['traits'], {'trait': 'value'})
self.assertEqual(msg['context']['library'], {
'name': 'analytics-python',
'version': VERSION
})
self.assertTrue(isinstance(msg['timestamp'], str))
self.assertEqual(msg['messageId'], 'messageId')
self.assertEqual(msg['distinct_id'], 'distinct_id')
self.assertEqual(msg['type'], 'identify')
def test_basic_group(self):
client = self.client
success, msg = client.group('distinct_id', 'groupId')
client.flush()
self.assertTrue(success)
self.assertFalse(self.failed)
self.assertEqual(msg['groupId'], 'groupId')
self.assertEqual(msg['distinct_id'], 'distinct_id')
self.assertEqual(msg['type'], 'group')
def test_advanced_group(self):
client = self.client
success, msg = client.group(
'distinct_id', 'groupId', {'trait': 'value'}, {'ip': '192.168.0.1'},
datetime(2014, 9, 3), 'messageId')
self.assertTrue(success)
self.assertEqual(msg['timestamp'], '2014-09-03T00:00:00+00:00')
self.assertEqual(msg['context']['ip'], '192.168.0.1')
self.assertEqual(msg['traits'], {'trait': 'value'})
self.assertEqual(msg['context']['library'], {
'name': 'analytics-python',
'version': VERSION
})
self.assertTrue(isinstance(msg['timestamp'], str))
self.assertEqual(msg['messageId'], 'messageId')
self.assertEqual(msg['distinct_id'], 'distinct_id')
self.assertEqual(msg['type'], 'group')
def test_basic_alias(self):
client = self.client
success, msg = client.alias('previousId', 'distinct_id')
client.flush()
self.assertTrue(success)
self.assertFalse(self.failed)
self.assertEqual(msg['previousId'], 'previousId')
self.assertEqual(msg['distinct_id'], 'distinct_id')
def test_basic_page(self):
client = self.client
success, msg = client.page('distinct_id', name='name')
self.assertFalse(self.failed)
client.flush()
self.assertTrue(success)
self.assertEqual(msg['distinct_id'], 'distinct_id')
self.assertEqual(msg['type'], 'page')
self.assertEqual(msg['name'], 'name')
def test_advanced_page(self):
client = self.client
success, msg = client.page(
'distinct_id', 'category', 'name', {'property': 'value'},
{'ip': '192.168.0.1'}, datetime(2014, 9, 3), 'messageId')
self.assertTrue(success)
self.assertEqual(msg['timestamp'], '2014-09-03T00:00:00+00:00')
self.assertEqual(msg['context']['ip'], '192.168.0.1')
self.assertEqual(msg['properties'], {'property': 'value'})
self.assertEqual(msg['context']['library'], {
'name': 'analytics-python',
'version': VERSION
})
self.assertEqual(msg['category'], 'category')
self.assertTrue(isinstance(msg['timestamp'], str))
self.assertEqual(msg['messageId'], 'messageId')
self.assertEqual(msg['distinct_id'], 'distinct_id')
self.assertEqual(msg['type'], 'page')
self.assertEqual(msg['name'], 'name')
def test_basic_screen(self):
client = self.client
success, msg = client.screen('distinct_id', name='name')
client.flush()
self.assertTrue(success)
self.assertEqual(msg['distinct_id'], 'distinct_id')
self.assertEqual(msg['type'], 'screen')
self.assertEqual(msg['name'], 'name')
def test_advanced_screen(self):
client = self.client
success, msg = client.screen(
'distinct_id', 'category', 'name', {'property': 'value'},
{'ip': '192.168.0.1'}, datetime(2014, 9, 3), 'messageId')
self.assertTrue(success)
self.assertEqual(msg['timestamp'], '2014-09-03T00:00:00+00:00')
self.assertEqual(msg['context']['ip'], '192.168.0.1')
self.assertEqual(msg['properties'], {'property': 'value'})
self.assertEqual(msg['context']['library'], {
'name': 'analytics-python',
'version': VERSION
})
self.assertTrue(isinstance(msg['timestamp'], str))
self.assertEqual(msg['messageId'], 'messageId')
self.assertEqual(msg['category'], 'category')
self.assertEqual(msg['distinct_id'], 'distinct_id')
self.assertEqual(msg['type'], 'screen')
self.assertEqual(msg['name'], 'name')
def test_flush(self):
client = self.client
# set up the consumer with more requests than a single batch will allow
for i in range(1000):
success, msg = client.identify('distinct_id', {'trait': 'value'})
# We can't reliably assert that the queue is non-empty here; that's
# a race condition. We do our best to load it up though.
client.flush()
# Make sure that the client queue is empty after flushing
self.assertTrue(client.queue.empty())
def test_shutdown(self):
client = self.client
# set up the consumer with more requests than a single batch will allow
for i in range(1000):
success, msg = client.identify('distinct_id', {'trait': 'value'})
client.shutdown()
# we expect two things after shutdown:
# 1. client queue is empty
# 2. consumer thread has stopped
self.assertTrue(client.queue.empty())
for consumer in client.consumers:
self.assertFalse(consumer.is_alive())
def test_synchronous(self):
client = Client('testsecret', sync_mode=True)
success, message = client.identify('distinct_id')
self.assertFalse(client.consumers)
self.assertTrue(client.queue.empty())
self.assertTrue(success)
def test_overflow(self):
client = Client('testsecret', max_queue_size=1)
# Ensure consumer thread is no longer uploading
client.join()
for i in range(10):
client.identify('distinct_id')
success, msg = client.identify('distinct_id')
# Make sure we are informed that the queue is at capacity
self.assertFalse(success)
def test_success_on_invalid_write_key(self):
client = Client('bad_key', on_error=self.fail)
client.track('distinct_id', 'event')
client.flush()
self.assertFalse(self.failed)
def test_unicode(self):
Client(six.u('unicode_key'))
def test_numeric_distinct_id(self):
self.client.track(1234, 'python event')
self.client.flush()
self.assertFalse(self.failed)
def test_debug(self):
Client('bad_key', debug=True)
def test_identify_with_date_object(self):
client = self.client
success, msg = client.identify(
'distinct_id',
{
'birthdate': date(1981, 2, 2),
},
)
client.flush()
self.assertTrue(success)
self.assertFalse(self.failed)
self.assertEqual(msg['traits'], {'birthdate': date(1981, 2, 2)})
def test_gzip(self):
client = Client('testsecret', on_error=self.fail, gzip=True)
for _ in range(10):
client.identify('distinct_id', {'trait': 'value'})
client.flush()
self.assertFalse(self.failed)
def test_user_defined_flush_at(self):
client = Client('testsecret', on_error=self.fail,
flush_at=10, flush_interval=3)
def mock_post_fn(*args, **kwargs):
self.assertEquals(len(kwargs['batch']), 10)
# the post function should be called 2 times, with a batch size of 10
# each time.
with mock.patch('analytics.consumer.post', side_effect=mock_post_fn) \
as mock_post:
for _ in range(20):
client.identify('distinct_id', {'trait': 'value'})
time.sleep(1)
self.assertEquals(mock_post.call_count, 2)
def test_user_defined_timeout(self):
client = Client('testsecret', timeout=10)
for consumer in client.consumers:
self.assertEquals(consumer.timeout, 10)
def test_default_timeout_15(self):
client = Client('testsecret')
for consumer in client.consumers:
self.assertEquals(consumer.timeout, 15)
+198
View File
@@ -0,0 +1,198 @@
import unittest
import mock
import time
import json
try:
from queue import Queue
except ImportError:
from Queue import Queue
from posthog.consumer import Consumer, MAX_MSG_SIZE
from posthogequest import APIError
class TestConsumer(unittest.TestCase):
def test_next(self):
q = Queue()
consumer = Consumer(q, '')
q.put(1)
next = consumer.next()
self.assertEqual(next, [1])
def test_next_limit(self):
q = Queue()
flush_at = 50
consumer = Consumer(q, '', flush_at)
for i in range(10000):
q.put(i)
next = consumer.next()
self.assertEqual(next, list(range(flush_at)))
def test_dropping_oversize_msg(self):
q = Queue()
consumer = Consumer(q, '')
oversize_msg = {'m': 'x' * MAX_MSG_SIZE}
q.put(oversize_msg)
next = consumer.next()
self.assertEqual(next, [])
self.assertTrue(q.empty())
def test_upload(self):
q = Queue()
consumer = Consumer(q, 'testsecret')
track = {
'type': 'track',
'event': 'python event',
'distinct_id': 'distinct_id'
}
q.put(track)
success = consumer.upload()
self.assertTrue(success)
def test_flush_interval(self):
# Put _n_ items in the queue, pausing a little bit more than
# _flush_interval_ after each one.
# The consumer should upload _n_ times.
q = Queue()
flush_interval = 0.3
consumer = Consumer(q, 'testsecret', flush_at=10,
flush_interval=flush_interval)
with mock.patch('analytics.consumer.post') as mock_post:
consumer.start()
for i in range(0, 3):
track = {
'type': 'track',
'event': 'python event %d' % i,
'distinct_id': 'distinct_id'
}
q.put(track)
time.sleep(flush_interval * 1.1)
self.assertEqual(mock_post.call_count, 3)
def test_multiple_uploads_per_interval(self):
# Put _flush_at*2_ items in the queue at once, then pause for
# _flush_interval_. The consumer should upload 2 times.
q = Queue()
flush_interval = 0.5
flush_at = 10
consumer = Consumer(q, 'testsecret', flush_at=flush_at,
flush_interval=flush_interval)
with mock.patch('analytics.consumer.post') as mock_post:
consumer.start()
for i in range(0, flush_at * 2):
track = {
'type': 'track',
'event': 'python event %d' % i,
'distinct_id': 'distinct_id'
}
q.put(track)
time.sleep(flush_interval * 1.1)
self.assertEqual(mock_post.call_count, 2)
def test_request(self):
consumer = Consumer(None, 'testsecret')
track = {
'type': 'track',
'event': 'python event',
'distinct_id': 'distinct_id'
}
consumer.request([track])
def _test_request_retry(self, consumer,
expected_exception, exception_count):
def mock_post(*args, **kwargs):
mock_post.call_count += 1
if mock_post.call_count <= exception_count:
raise expected_exception
mock_post.call_count = 0
with mock.patch('analytics.consumer.post',
mock.Mock(side_effect=mock_post)):
track = {
'type': 'track',
'event': 'python event',
'distinct_id': 'distinct_id'
}
# request() should succeed if the number of exceptions raised is
# less than the retries paramater.
if exception_count <= consumer.retries:
consumer.request([track])
else:
# if exceptions are raised more times than the retries
# parameter, we expect the exception to be returned to
# the caller.
try:
consumer.request([track])
except type(expected_exception) as exc:
self.assertEqual(exc, expected_exception)
else:
self.fail(
"request() should raise an exception if still failing "
"after %d retries" % consumer.retries)
def test_request_retry(self):
# we should retry on general errors
consumer = Consumer(None, 'testsecret')
self._test_request_retry(consumer, Exception('generic exception'), 2)
# we should retry on server errors
consumer = Consumer(None, 'testsecret')
self._test_request_retry(consumer, APIError(
500, 'code', 'Internal Server Error'), 2)
# we should retry on HTTP 429 errors
consumer = Consumer(None, 'testsecret')
self._test_request_retry(consumer, APIError(
429, 'code', 'Too Many Requests'), 2)
# we should NOT retry on other client errors
consumer = Consumer(None, 'testsecret')
api_error = APIError(400, 'code', 'Client Errors')
try:
self._test_request_retry(consumer, api_error, 1)
except APIError:
pass
else:
self.fail('request() should not retry on client errors')
# test for number of exceptions raise > retries value
consumer = Consumer(None, 'testsecret', retries=3)
self._test_request_retry(consumer, APIError(
500, 'code', 'Internal Server Error'), 3)
def test_pause(self):
consumer = Consumer(None, 'testsecret')
consumer.pause()
self.assertFalse(consumer.running)
def test_max_batch_size(self):
q = Queue()
consumer = Consumer(
q, 'testsecret', flush_at=100000, flush_interval=3)
track = {
'type': 'track',
'event': 'python event',
'distinct_id': 'distinct_id'
}
msg_size = len(json.dumps(track).encode())
# number of messages in a maximum-size batch
n_msgs = int(475000 / msg_size)
def mock_post_fn(_, data, **kwargs):
res = mock.Mock()
res.status_code = 200
self.assertTrue(len(data.encode()) < 500000,
'batch size (%d) exceeds 500KB limit'
% len(data.encode()))
return res
with mock.patch('analytics.request._session.post',
side_effect=mock_post_fn) as mock_post:
consumer.start()
for _ in range(0, n_msgs + 2):
q.put(track)
q.join()
self.assertEquals(mock_post.call_count, 2)
+49
View File
@@ -0,0 +1,49 @@
import unittest
import analytics
class TestModule(unittest.TestCase):
def failed(self):
self.failed = True
def setUp(self):
self.failed = False
analytics.write_key = 'testsecret'
analytics.on_error = self.failed
def test_no_write_key(self):
analytics.write_key = None
self.assertRaises(Exception, analytics.track)
def test_no_host(self):
analytics.host = None
self.assertRaises(Exception, analytics.track)
def test_track(self):
analytics.track('distinct_id', 'python module event')
analytics.flush()
def test_identify(self):
analytics.identify('distinct_id', {'email': 'user@email.com'})
analytics.flush()
def test_group(self):
analytics.group('distinct_id', 'groupId')
analytics.flush()
def test_alias(self):
analytics.alias('previousId', 'distinct_id')
analytics.flush()
def test_page(self):
analytics.page('distinct_id')
analytics.flush()
def test_screen(self):
analytics.screen('distinct_id')
analytics.flush()
def test_flush(self):
analytics.flush()
+53
View File
@@ -0,0 +1,53 @@
from datetime import datetime, date
import unittest
import json
import requests
from posthogequest import post, DatetimeSerializer
class TestRequests(unittest.TestCase):
def test_valid_request(self):
res = post(batch=[{
'distinct_id': 'distinct_id',
'event': 'python event',
'type': 'track'
}])
self.assertEqual(res.status_code, 200)
def test_invalid_request_error(self):
self.assertRaises(Exception, post, 'testsecret',
'https://t.posthog.com', False, '[{]')
def test_invalid_host(self):
self.assertRaises(Exception, post, 'testsecret',
't.posthog.com/', batch=[])
def test_datetime_serialization(self):
data = {'created': datetime(2012, 3, 4, 5, 6, 7, 891011)}
result = json.dumps(data, cls=DatetimeSerializer)
self.assertEqual(result, '{"created": "2012-03-04T05:06:07.891011"}')
def test_date_serialization(self):
today = date.today()
data = {'created': today}
result = json.dumps(data, cls=DatetimeSerializer)
expected = '{"created": "%s"}' % today.isoformat()
self.assertEqual(result, expected)
def test_should_not_timeout(self):
res = post(batch=[{
'distinct_id': 'distinct_id',
'event': 'python event',
'type': 'track'
}], timeout=15)
self.assertEqual(res.status_code, 200)
def test_should_timeout(self):
with self.assertRaises(requests.ReadTimeout):
post(batch=[{
'distinct_id': 'distinct_id',
'event': 'python event',
'type': 'track'
}], timeout=0.0001)
+78
View File
@@ -0,0 +1,78 @@
from datetime import date, datetime, timedelta
from decimal import Decimal
import unittest
from dateutil.tz import tzutc
import six
from posthog import utils
class TestUtils(unittest.TestCase):
def test_timezone_utils(self):
now = datetime.now()
utcnow = datetime.now(tz=tzutc())
self.assertTrue(utils.is_naive(now))
self.assertFalse(utils.is_naive(utcnow))
fixed = utils.guess_timezone(now)
self.assertFalse(utils.is_naive(fixed))
shouldnt_be_edited = utils.guess_timezone(utcnow)
self.assertEqual(utcnow, shouldnt_be_edited)
def test_clean(self):
simple = {
'decimal': Decimal('0.142857'),
'unicode': six.u('woo'),
'date': datetime.now(),
'long': 200000000,
'integer': 1,
'float': 2.0,
'bool': True,
'str': 'woo',
'none': None
}
complicated = {
'exception': Exception('This should show up'),
'timedelta': timedelta(microseconds=20),
'list': [1, 2, 3]
}
combined = dict(simple.items())
combined.update(complicated.items())
pre_clean_keys = combined.keys()
utils.clean(combined)
self.assertEqual(combined.keys(), pre_clean_keys)
def test_clean_with_dates(self):
dict_with_dates = {
'birthdate': date(1980, 1, 1),
'registration': datetime.utcnow(),
}
self.assertEqual(dict_with_dates, utils.clean(dict_with_dates))
def test_bytes(self):
if six.PY3:
item = bytes(10)
else:
item = bytearray(10)
utils.clean(item)
def test_clean_fn(self):
cleaned = utils.clean({'fn': lambda x: x, 'number': 4})
self.assertEqual(cleaned['number'], 4)
# TODO: fixme, different behavior on python 2 and 3
if 'fn' in cleaned:
self.assertEqual(cleaned['fn'], None)
def test_remove_slash(self):
self.assertEqual('http://posthog.io',
utils.remove_trailing_slash('http://posthog.io/'))
self.assertEqual('http://posthog.io',
utils.remove_trailing_slash('http://posthog.io'))
+87
View File
@@ -0,0 +1,87 @@
from dateutil.tz import tzlocal, tzutc
from datetime import date, datetime
from decimal import Decimal
import logging
import numbers
import six
log = logging.getLogger('posthog')
def is_naive(dt):
"""Determines if a given datetime.datetime is naive."""
return dt.tzinfo is None or dt.tzinfo.utcoffset(dt) is None
def total_seconds(delta):
"""Determines total seconds with python < 2.7 compat."""
# http://stackoverflow.com/questions/3694835/python-2-6-5-divide-timedelta-with-timedelta
return (delta.microseconds
+ (delta.seconds + delta.days * 24 * 3600) * 1e6) / 1e6
def guess_timezone(dt):
"""Attempts to convert a naive datetime to an aware datetime."""
if is_naive(dt):
# attempts to guess the datetime.datetime.now() local timezone
# case, and then defaults to utc
delta = datetime.now() - dt
if total_seconds(delta) < 5:
# this was created using datetime.datetime.now()
# so we are in the local timezone
return dt.replace(tzinfo=tzlocal())
else:
# at this point, the best we can do is guess UTC
return dt.replace(tzinfo=tzutc())
return dt
def remove_trailing_slash(host):
if host.endswith('/'):
return host[:-1]
return host
def clean(item):
if isinstance(item, Decimal):
return float(item)
elif isinstance(item, (six.string_types, bool, numbers.Number, datetime,
date, type(None))):
return item
elif isinstance(item, (set, list, tuple)):
return _clean_list(item)
elif isinstance(item, dict):
return _clean_dict(item)
else:
return _coerce_unicode(item)
def _clean_list(list_):
return [clean(item) for item in list_]
def _clean_dict(dict_):
data = {}
for k, v in six.iteritems(dict_):
try:
data[k] = clean(v)
except TypeError:
log.warning(
'Dictionary values must be serializeable to '
'JSON "%s" value %s of type %s is unsupported.',
k, v, type(v),
)
return data
def _coerce_unicode(cmplx):
try:
item = cmplx.decode("utf-8", "strict")
except AttributeError as exception:
item = ":".join(exception)
item.decode("utf-8", "strict")
log.warning('Error decoding: %s', item)
return None
return item
+1
View File
@@ -0,0 +1 @@
VERSION = '1.0.0'
+2
View File
@@ -0,0 +1,2 @@
[bdist_wheel]
universal = 1
+63
View File
@@ -0,0 +1,63 @@
import os
import sys
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
# Don't import analytics-python module here, since deps may not be installed
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'posthog'))
from version import VERSION
long_description = '''
PostHog is developer-friendly, self-hosted product analytics. posthog-python is the python package.
'''
install_requires = [
"requests>=2.7,<3.0",
"six>=1.5",
"monotonic>=1.5",
"backoff==1.6.0",
"python-dateutil>2.1"
]
tests_require = [
"mock>=2.0.0"
]
setup(
name='posthog',
version=VERSION,
url='https://github.com/posthog/posthog-python',
author='Posthog',
author_email='hey@posthog.com',
maintainer='PostHog',
maintainer_email='hey@posthog.com',
test_suite='posthog.test.all',
packages=['posthog', 'posthog.test'],
license='MIT License',
install_requires=install_requires,
tests_require=tests_require,
description='Integrate PostHog into any python application.',
long_description=long_description,
classifiers=[
"Development Status :: 5 - Production/Stable",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Programming Language :: Python",
"Programming Language :: Python :: 2",
"Programming Language :: Python :: 2.6",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.2",
"Programming Language :: Python :: 3.3",
"Programming Language :: Python :: 3.4",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
],
)
+99
View File
@@ -0,0 +1,99 @@
import analytics
import argparse
import json
import logging
__name__ = 'simulator.py'
__version__ = '0.0.1'
__description__ = 'scripting simulator'
def json_hash(str):
if str:
return json.loads(str)
# analytics -method=<method> -posthog-write-key=<posthogWriteKey> [options]
parser = argparse.ArgumentParser(description='send a posthog message')
parser.add_argument('--writeKey', help='the posthog writeKey')
parser.add_argument('--type', help='The posthog message type')
parser.add_argument('--distinct_id', help='the user id to send the event as')
parser.add_argument(
'--anonymousId', help='the anonymous user id to send the event as')
parser.add_argument(
'--context', help='additional context for the event (JSON-encoded)')
parser.add_argument('--event', help='the event name to send with the event')
parser.add_argument(
'--properties', help='the event properties to send (JSON-encoded)')
parser.add_argument(
'--name', help='name of the screen or page to send with the message')
parser.add_argument(
'--traits', help='the identify/group traits to send (JSON-encoded)')
parser.add_argument('--groupId', help='the group id')
options = parser.parse_args()
def failed(status, msg):
raise Exception(msg)
def track():
analytics.track(options.distinct_id, options.event, anonymous_id=options.anonymousId,
properties=json_hash(options.properties), context=json_hash(options.context))
def page():
analytics.page(options.distinct_id, name=options.name, anonymous_id=options.anonymousId,
properties=json_hash(options.properties), context=json_hash(options.context))
def screen():
analytics.screen(options.distinct_id, name=options.name, anonymous_id=options.anonymousId,
properties=json_hash(options.properties), context=json_hash(options.context))
def identify():
analytics.identify(options.distinct_id, anonymous_id=options.anonymousId,
traits=json_hash(options.traits), context=json_hash(options.context))
def group():
analytics.group(options.distinct_id, options.groupId, json_hash(options.traits),
json_hash(options.context), anonymous_id=options.anonymousId)
def unknown():
print()
analytics.write_key = options.writeKey
analytics.on_error = failed
analytics.debug = True
log = logging.getLogger('posthog')
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
log.addHandler(ch)
switcher = {
"track": track,
"page": page,
"screen": screen,
"identify": identify,
"group": group
}
func = switcher.get(options.type)
if func:
func()
analytics.shutdown()
else:
print("Invalid Message Type " + options.type)