Skip to content

Instantly share code, notes, and snippets.

@Nihhaar

Nihhaar/logger.py

Created August 27, 2021 04:20
Show Gist options
  • Save Nihhaar/83115935c655cdd5065bf18f98f9a6a4 to your computer and use it in GitHub Desktop.
Save Nihhaar/83115935c655cdd5065bf18f98f9a6a4 to your computer and use it in GitHub Desktop.
Download ZIP
PySpark logging
Raw
logger.py
import os
import logging
import sys
class YarnLogger:
@staticmethod
def setup_logger():
if not 'LOG_DIRS' in os.environ:
sys.stderr.write('Missing LOG_DIRS environment variable, pyspark logging disabled')
return
file = os.environ['LOG_DIRS'].split(',')[0] + '/pyspark.log'
logging.basicConfig(filename=file, level=logging.INFO,
format='%(asctime)s.%(msecs)03d %(levelname)s %(module)s - %(funcName)s: %(message)s')
def __getattr__(self, key):
return getattr(logging, key)
YarnLogger.setup_logger()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment