Envision performance tuning

[JVillella]

Envision cost seems to be about 40 fps on a regularly 100 fps run. That should turn out to be about 7 milliseconds.

We want to ideally cut that down to 3 or less.

[liamchzh]

#451 (comment)

Are we are sending envision state asynchronously?

@Gamenot @JingfeiPeng We need to figure this out.

[JingfeiPeng]

https://github.com/huawei-noah/SMARTS/blob/develop/smarts/core/smarts.py#L948
The only difference between headless True and False is this emit envision state call and self._envision.send(state) here. If I comment out self._envision.send(state) and set Headless to False, Smarts does have the same performance as when headless is True.

when self._envision.send(state) is not commented out and headless is false:

episode time: 8.354718208s
episode time: 35.294245005s
episode time: 5.584953308s
episode time: 7.116137743s
episode time: 0.467200994s
episode time: 29.39287138s
episode time: 15.097165346s
episode time: 22.978841542999998s
episode time: 14.170237779999999s
episode time: 34.543913603s
total time spent in _try_emit_envision_state: 1.0054428669999993s
total time spent in all episodes: 173.00028490999998s

when self._envision.send(state) is commented out and headless is false:

episode time: 4.792479992s
episode time: 13.881719112s
episode time: 3.1874554159999997s
episode time: 3.619199038s
episode time: 0.269412518s
episode time: 11.487694025000001s
episode time: 7.208214045s
episode time: 10.05633688s
episode time: 6.886660576000001s
episode time: 13.744481563999999s
total time spent in _try_emit_envision_state: 0.8824660859999996s
total time spent in all episodes: 75.133653166s

The total time spent on _try_emit_envision_state is roughly the same between the two, but the total time spent on all the episodes(total time spent for running the example) has a large difference between the two ways

[JingfeiPeng]

@Adaickalavan A script to calculate total time from smarts logs

import fileinput

def cal_time(fileName):
    with fileinput.input(files=(fileName)) as f:
        total = float(0)
        totalExampleRunning = float(0)
        for line in f:
            if "Emit envision" in line:
                time = float(line.split("took: ")[1].split('ms')[0])
                total += time
            elif "run example" in line:
                time = float(line.split("took: ")[1].split('ms')[0])
                totalExampleRunning += time
                print(f"episode time: {time/1000}s")
        print(f"total time spent in _try_emit_envision_state: {total/1000}s")
        print(f"total time spent in all episodes: {totalExampleRunning/1000}s")

cal_time('logtimeit.txt')
cal_time('logtimeitNotSend.txt')