Kyle Edwards

Miscellaneous Notes

#programming #python

While there’s no replacement for reading the official Python documentation, below is a collection of notes around the features and philosophies of the Python language, especially those that set it apart from other languages.

The Data Model (Link)

Python handles allocating and freeing memory mainly with reference counting, however it uses garbage collection to occasionally clear cyclic references. You can use id() to retrieve the memory location for a piece of data.

Since all variables are objects, Python gives you access to their properties. User-defined functions allow you to access both immutable and mutable properties like __doc__, __name__, __defaults__, __globals__, __annotations__, and more.

Namespaces

Classes and modules have a concept of namespacing, which is how property lookups occur.

When you attempt to access an attribute from a class instance, the interpreter first attempts to retrieve that attribute from the instance itself. If it’s not present on the instance, it then checks the class attributes before raising an AttributeError. An important thing to note here is that using a class attribute as a default value may be dangerous because anyone could overwrite this at the class level and cause unintended effects for instances.

class Element:
    _default_value = False

    def set_value(self, overwrite):
        self._default_value = overwrite

a = Element()
b = Element()

print(Element._default_value)
# False

print(a._default_value)
# False

b.set_value(True)
print(b._default_value)
# True

Element._default_value = True
print(a._default_value)
# True

Code Objects and co_flags

There’s a ton of possibly useful information that can be gathered from a function’s __code__ object. The co_flags property is a set of bit flags that inform the interpreter if the function uses arbitrary arguments, keyword arguments, is a generator, etc…

Protocols and Hooks

While Python is a language with traditional object-oriented features, idiomatic Python relies on duck-typing in favor of inheritance. As long as user-defined classes implement the right “dunder” methods (or protocols), you can tap into power built-in functions, or more importantly, build out reliable contracts between module authors and their users. It’s possible to take these steps further by using metaclasses or abstract base classes to enforce these contracts much earlier on class definition, avoiding annoying runtime exceptions.

Introspection

Introspection can provide useful information about classes (inspect.getmembers()) and functions (inspect.getfullargspec()) that can be used for creative (and usually opaque) purposes. You could use the inspect module within decorators to analyze the arguments of a wrapped function. In fact, FastAPI provides a clever pattern that uses inspect to repurpose argument defaults as a Dependency Injection mechanism. (See the __build_class__ for interesting ways to use introspection.)

Metaclasses

Metaclasses can be used to inspect and hook into subclass instantiation at runtime and enforce certain constraints. The abc library implements an Abstract Base Class metaclass already does this.

Metaclasses are used to spy on subclass attribute names and use them in novel ways. ORMs libraries tend to do this to generate queries, do validation, etc… (See __init_subclass__.)

Frequently, metaclasses are used like class decorators but have better inheritance support when using a decorated class as a superclass.

Interning

To save memory, the Python interpreter reuses small strings and integers. You can observe this with the following behavior:

a0 = 255
a1 = 255
a0 is a1 # True

b0 = 2500000
b1 = 2500000
b0 is b1 # False

Useful Decorators

# This is drastically oversimplified but I was really
# curious as to how FastAPI handles dependency injection
# internally.
#
# This doesn't handle any of the fun stuff with async,
# running things in the threadpool, handling recursive
# sub-dependencies, etc.

from typing import Optional, Callable, Dict, Any
import pprint
import inspect


class Depends:
    def __init__(self, dependency: Optional[Callable]):
        self.dependency = dependency


def get_typed_annotation(param: inspect.Parameter, globalns: Dict[str, Any]) -> Any:
    annotation = param.annotation
    if isinstance(annotation, str):
        # Temporary ignore type
        # Ref: https://github.com/samuelcolvin/pydantic/issues/1738
        annotation = ForwardRef(annotation)  # type: ignore
        annotation = evaluate_forwardref(annotation, globalns, globalns)
    return annotation


def get_typed_signature(call: Callable) -> inspect.Signature:
    signature = inspect.signature(call)
    globalns = getattr(call, "__globals__", {})
    typed_params = [
        inspect.Parameter(
            name=param.name,
            kind=param.kind,
            default=param.default,
            annotation=get_typed_annotation(param, globalns),
        )
        for param in signature.parameters.values()
    ]
    typed_signature = inspect.Signature(typed_params)
    return typed_signature


def dependency_injection(*args, **kwargs):
    pprint.pprint(args)
    pprint.pprint(kwargs)
    return True


def inject(func):
    def wrapper(*args, **kwargs):
        print("Injecting dependency if possible...")
        endpoint_signature = get_typed_signature(func)
        signature_params = endpoint_signature.parameters
        all_args = inspect.getfullargspec(func)
        print(all_args)
        new_args = []
        new_kwargs = {}
        for param_name, param in signature_params.items():
            # print(param_name, param)
            if isinstance(param.default, Depends):
                res = param.default.dependency(*args, **kwargs)
                if param_name in args:
                    idx = all_args.args.index(param_name)
                    new_args[idx] = res
                else:
                    new_kwargs[param_name] = res
            else:
                if param_name in all_args.args:
                    idx = all_args.args.index(param_name)
                    new_args[idx] = args[idx]
                else:
                    new_kwargs[param_name] = kwargs[param_name]
        func(*new_args, **new_kwargs)
        print("After inner function is called...")

    return wrapper


@inject
def di_test(success: bool = Depends(dependency_injection)):
    print(f"Dependency injected: {success}")


di_test(1, 2)

Complex Numbers

Python natively supports complex numbers. You can use j to define imaginary literals (like 3 + 4j).

Confusing Behaviors

Mutable Types as Default Parameters

This is a bad idea because the default parameter is shared across all calls of the function. Any change to this default value will be present on subsequent calls.

def with_defaults(x, lst = []):
    lst.append(x)
    return lst

with_defaults(1) # [1]
with_defaults(2) # [1, 2]
with_defaults(3, []) # [3]; This is fine
with_defaults(4) # [1, 2, 4]

Returning from try/finally Blocks

This is not behavior specifically confined to Python, however it’s generally considered good practice to not handle any sort of control flow within a finally block.

The return value of a function is determined by the last return statement executed. Since the finally clause always executes, a return statement executed in the finally clause will always be the last one executed… (Source)

def ignore_my_return_why_dont_you():
    try:
        return True
    finally:
        return False

ignore_my_return_why_dont_you()
# False

Pattern Matching

Python introduced a match/case pattern matching syntax in version 3.10. See PEPs 634, 635, and 636 for more info.

Additional Resources