Skip to content

Home

Jump to bottom Edit New page
Juan Carlos edited this page Dec 19, 2019 · 18 revisions

Welcome to the nimpy wiki!.

FAQ

Publish to PYPI

Create a new file on your repo named setup.cfg, and paste this on that new file:

# See: https://setuptools.readthedocs.io/en/latest/setuptools.html#metadata
[metadata]
name             = example
provides         = example
description      = example package
url              = https://github.com/example/example
download_url     = https://github.com/example/example
author           = Graydon Hoare
author_email     = [email protected]
maintainer       = Ryan Dahl
maintainer_email = [email protected]
keywords         = python3, exampletag, sometag
license          = MIT
platforms        = Linux, Darwin, Windows
version          = attr: somepythonmodule.__version__
project_urls     =
    Docs = https://github.com/example/example/README.md
    Bugs = https://github.com/example/example/issues
    CI   = https://github.com/example/example/actions

license_file = LICENSE
long_description = file: README.md
long_description_content_type = text/markdown
classifiers =  # https://pypi.python.org/pypi?%3Aaction=list_classifiers
    Development Status :: 5 - Production/Stable
    Environment :: Console
    Environment :: Other Environment
    Intended Audience :: Developers
    Intended Audience :: Other Audience
    Natural Language :: English
    License :: OSI Approved :: GNU General Public License (GPL)
    License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)
    License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)
    License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
    Operating System :: OS Independent
    Operating System :: POSIX :: Linux
    Operating System :: Microsoft :: Windows
    Operating System :: MacOS :: MacOS X
    Programming Language :: Python
    Programming Language :: Python :: 3
    Programming Language :: Python :: 3 :: Only
    Programming Language :: Python :: 3.6
    Programming Language :: Python :: Implementation :: CPython
    Topic :: Software Development

[options]
zip_safe = True
include_package_data = True
python_requires  = >3.6
packages         = find:

[bdist_wheel]
universal = 1

[install_lib]
compile = 0
optimize = 2

[bdist_egg]
exclude-source-files = false

[options.package_data]
* = *.c, *.h, nimbase.h

[options.exclude_package_data]
* = *.py, *.nim, *.so, *.md, *.dll, *.zip, *.js, *.tests, *.tests.*, tests.*, tests, README.md

[options.packages.find]
where   = .
include = *.c, *.h, nimbase.h
exclude = *.py, *.nim, *.so, *.dll, *.zip, *.js, *.tests, *.tests.*, tests.*, tests, README.md

Edit the setup.cfg from placeholder values to your personal settings. This setup.cfg is like a Template that you should edit for it to work correctly. Check the Python Documentation if you dont know how to make a setup.cfg.

Create a new file on your repo named setup.py, and paste this on that new file:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from setuptools import Extension
from setuptools import setup
from os import listdir

sources = []
for c_source_file in listdir():
    if c_source_file.endswith(".c"):
        sources.append(c_source_file)

setup(
    ext_modules = [
        Extension(
            name               = "example",  # Change it to your project name.
            sources            = sources,
            extra_compile_args = ["-flto", "-ffast-math", "-march=native", "-mtune=native", "-O3", "-fno-ident", "-fsingle-precision-constant"],  # Change it to your project needs.
            extra_link_args    = ["-s"],
            include_dirs       = ["."],
        )
    ]
)

Edit the setup.py from placeholder values to your personal settings, you must edit the name and extra_compile_args at least. This setup.py is like a Template that you should edit for it to work correctly. Check the Python Documentation if you dont know how to make a setup.py.

Commit and Push the setup.py and setup.cfg to GitHub. From now on, this documentation assumes you already have a working setup.py and setup.cfg on your Repo. If you have problems creating the setup.py and setup.cfg check the Python Documentation.

You must have a valid active PyPI username and password to upload to PyPI, if you do not have one go to https://pypi.org/account/register and register yourself, if you do have one go to https://pypi.org/account/login and login to check if its working.

Go to https://github.com/USER/REPO/settings/actions, where USER is your GitHub username, and REPO is your repo, check that GitHub Actions must be Enabled.

Go to https://github.com/USER/REPO/settings/secrets/new, where USER is your GitHub username, and REPO is your repo.

Create 2 new Secrets named PYPI_USERNAME and PYPI_PASSWORD, where PYPI_USERNAME is your PyPI username, and PYPI_PASSWORD is your PyPI password, Secrets wont need quotes, dont worry both will be Encrypted, not visible from the web, nor visible from Forks.

On your Repo create a new file /.github/workflows/nimpy_pypi_upload.yml, create the folders if needed, create the file if needed, and paste the whole following content:

name: Upload to PYPI
on:
  release:
    types: [created]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - uses: actions/setup-python@v1
    - name: Set Global Environment Variables
      uses: allenevans/[email protected]
      with:
        CHOOSENIM_CHOOSE_VERSION: "1.0.4"
        CHOOSENIM_NO_ANALYTICS: 1
        TWINE_NON_INTERACTIVE: 1
        TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}   # https://github.com/USER/REPO/settings/secrets/new
        TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
        MAIN_MODULE: "src/main.nim"
        #TWINE_REPOSITORY_URL: "https://test.pypi.org/legacy/"  # Upload to PYPI Testing fake server.
        #TWINE_REPOSITORY: "https://test.pypi.org/legacy/"
    - name: Update Python PIP
      run: pip3 install --upgrade --disable-pip-version-check pip setuptools twine
    - name: Cache choosenim
      id: cache-choosenim
      uses: actions/cache@v1
      with:
        path: ~/.choosenim
        key: ${{ runner.os }}-choosenim-$CHOOSENIM_CHOOSE_VERSION
    - name: Cache nimble
      id: cache-nimble
      uses: actions/cache@v1
      with:
        path: ~/.nimble
        key: ${{ runner.os }}-nimble-$CHOOSENIM_CHOOSE_VERSION
    - name: Install Nim via Choosenim
      if: steps.cache-choosenim.outputs.cache-hit != 'true' || steps.cache-nimble.outputs.cache-hit != 'true'
      run: |
        curl https://nim-lang.org/choosenim/init.sh -sSf > init.sh
        sh init.sh -y
    - name: Nimble Refresh
      run: |
        export PATH=$HOME/.nimble/bin:$PATH
        nimble -y refresh
    - name: Nimble Install dependencies
      run: |
        export PATH=$HOME/.nimble/bin:$PATH
        nimble -y install nimpy
    - name: Prepare Files
      run: |
        mkdir --verbose --parents dist/
        rm --verbose --force --recursive *.c *.h *.so *.pyd *.egg-info/ dist/*.zip
        cp --verbose --force ~/.choosenim/toolchains/nim-$CHOOSENIM_CHOOSE_VERSION/lib/nimbase.h nimbase.h
    - name: Compile to C
      run: |
        export PATH=$HOME/.nimble/bin:$PATH
        nim compileToC --compileOnly -d:release -d:danger -d:ssl --threads:on --app:lib --opt:speed --gc:markAndSweep --nimcache:. $MAIN_MODULE
    - name: Publish to PYPI
      run: |
        python3 setup.py --verbose sdist --formats=zip
        rm --verbose --force --recursive *.c *.h *.so *.pyd *.egg-info/
        twine upload --verbose --disable-progress-bar dist/*.zip

Edit CHOOSENIM_CHOOSE_VERSION to the version you choose or the latest. Edit MAIN_MODULE: "src/main.nim" to your main module .nim file. You can alternatively compile to C using NimScript, Nimble build tasks, Bash Scripts, etc.

If you want to upload to the Testing fake PyPI Server instead of the real one, uncomment the lines:

TWINE_REPOSITORY_URL: "https://test.pypi.org/legacy/"  # Upload to PYPI Testing fake server.
TWINE_REPOSITORY: "https://test.pypi.org/legacy/"

Commit and Push the YAML to GitHub.

Go to https://github.com/USER/REPO/releases/new, where USER is your GitHub username, and REPO is your repo, create a new Release to trigger the new GitHub Action of the YAML, editing an existing release wont work, so the release must be a new one.

Wait for the GitHub Action to complete, you can check the progress at https://github.com/USER/REPO/actions, if the GitHub Actions is sucessful then you should have your project uploaded to PyPI.

Remember that PyPI wont allow to re-upload the same file even if you delete it, so if you want to overwrite the file uploaded to PyPI you must bump version up.

From now on, every time you make a new Release, it will be automatically uploaded to PYPI.

Live Working Example: https://github.com/juancarlospaco/faster-than-requests/commit/a4be5be5ef7dfbe110df882a9fe45cfb869ff336/checks?check_suite_id=336800193

Build every Push

On your Repo create a new file /.github/workflows/nimpy_build.yml, create the folders if needed, create the file if needed, and paste the whole following content:

name: Build Nimpy

on: [push]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v1
    - uses: actions/setup-python@v1

    - name: Set Global Environment Variables
      uses: allenevans/[email protected]
      with:
        CHOOSENIM_CHOOSE_VERSION: "1.0.4"
        CHOOSENIM_NO_ANALYTICS: 1
        MAIN_MODULE: "src/main.nim"
        MODULE_NAME: "example"

    - name: Update Python PIP
      run: pip3 install --upgrade --disable-pip-version-check pip setuptools

    - name: Cache choosenim
      id: cache-choosenim
      uses: actions/cache@v1
      with:
        path: ~/.choosenim
        key: ${{ runner.os }}-choosenim-$CHOOSENIM_CHOOSE_VERSION

    - name: Cache nimble
      id: cache-nimble
      uses: actions/cache@v1
      with:
        path: ~/.nimble
        key: ${{ runner.os }}-nimble-$CHOOSENIM_CHOOSE_VERSION

    - name: Install Nim via Choosenim
      if: steps.cache-choosenim.outputs.cache-hit != 'true' || steps.cache-nimble.outputs.cache-hit != 'true'
      run: |
        curl https://nim-lang.org/choosenim/init.sh -sSf > init.sh
        sh init.sh -y

    - name: Nimble Refresh
      run: |
        export PATH=$HOME/.nimble/bin:$PATH
        nimble -y refresh

    - name: Nimble Install dependencies
      run: |
        export PATH=$HOME/.nimble/bin:$PATH
        nimble -y install nimpy

    - name: Compile DEBUG Mode
      run: |
        export PATH=$HOME/.nimble/bin:$PATH
        nim c -d:ssl --threads:on --app:lib $MAIN_MODULE

    - name: Compile RELEASE Mode
      run: |
        export PATH=$HOME/.nimble/bin:$PATH
        nim c -d:release -d:danger -d:ssl --threads:on --app:lib -o:$MODULE_NAME.so $MAIN_MODULE

    - name: Strip Binary
      run: strip --strip-all $MODULE_NAME.so

    - name: Import Binary
      run: python3 -c "import $MODULE_NAME"  # You can add more steps and tests after this step.

Edit CHOOSENIM_CHOOSE_VERSION to the version you choose or the latest. Edit MAIN_MODULE: "src/main.nim" to your main module .nim file. Edit MODULE_NAME: "example" to your actual module Python name. You can alternatively compile to C using NimScript, Nimble build tasks, Bash Scripts, etc.

Commit and Push the YAML to GitHub.

Wait for the GitHub Action to complete, you can check the progress at https://github.com/USER/REPO/actions.

From now on, every time you make push a new commit, it will be automatically Build using GitHub Actions.

Live Working Example: https://github.com/juancarlospaco/faster-than-requests/commit/64424826ac9d23c2ded3a5fcd5c6ced45d5d3c63/checks?check_suite_id=336794312

Pragmas

Useful Pragmas that work with Nimpy.

discardable

Syntax {.discardable.}

Docs https://nim-lang.org/docs/manual.html#statements-and-expressions-discard-statement

noreturn

Syntax {.noreturn.}

Docs https://nim-lang.org/docs/manual.html#pragmas-noreturn-pragma

Mark functions that do not need a return value with this pragma, the generated machine code will not have the return.

compiletime

Syntax {.compiletime.}

Docs https://nim-lang.org/docs/manual.html#pragmas-compiletime-pragma

If you want to get truly immutable things on Python or if you want to get a compile time function execution, you can use compiletime pragma or simply return a const.

linearscanend

Syntax {.linearscanend.}

Docs https://nim-lang.org/docs/manual.html#pragmas-linearscanend-pragma

If you come from Python, on Nim you have a better and faster alternative to if..elif...else, the case switch compiles to more optimized code (all cases must be covered), to improve performance of a long case switch you can use linearscanend pragma.

exportc

Syntax {.exportc.}

Docs https://nim-lang.org/docs/manual.html#foreign-function-interface-exportc-pragma

C and C++ by design use Mangling, in short if your code has foo() function then it will compile to foo_s0m3H4sh(), this is not invented by Nim is just how things work, if you want to debug/read the C code by hand, you can use exportc pragma to temporarily omit the mangling. This is not recommended for production.

proc foo() {.exportpy, exportc: "foo".} =

Other Resources

Import Nim source code files on Python

With Nimporter, you can simply import Nim source code files as if they were Python modules, and use them seamlessly with Python code. Compile Nim extensions when importing them automatically!. You can simply import the nim source code files.

Python Syntax

Nimpylib is a collection of python-like operators and functions (syntax sugar). It can help you to translate your Python program to Nim.

PIP Alternative

Alternative PIP re-implementation written using Nim, uses official PyPI API, faster and smaller, wont depend on Pythons PIP so it works even with PIP/Virtualenv broken, designed to install Python stuff on Docker/Alpine.

Nim for Python developers

Official Wiki for Python developers learning Nim.

Install Nim inside Virtualenv

This package can be used to install the Nim compiler inside a Python virtualenv. PIP installable.

Add a custom footer
Add a custom sidebar
Clone this wiki locally