Posted on | In

Installation:

  • Ubuntu

    1
    2
    sudo apt-get install texlive-full	
    sudo apt-get install texmaker

  • Windows

    1. Install Miktex: https://miktex.org/download
    2. Install texmaker or texstudio

Extension:

  1. Latexdiff

Math input:

  1. Include the definition.tex file with macro definition. \input{definition} in the main text.

  2. Mathpix Snip can convert images to LaTeX.

  3. IguanaTex allows you to insert LaTeX formulations in PowerPoint.

Tips:

  1. Refer to this for commonly used symbols

  2. Set color: \usepackage{xcolor} and \textcolor{red}{}

  3. Setlength: set the indentation in paragraphs \setlength\parindent{0pt}; set the separation between paragraphs \setlength{\parskip}{10pt}; set column width in table \setlength{\tabcolsep}{12pt}; set the space below table/figure \setlength{\textfloatsep}{2pt}

  4. Math annotation: expectation \usepackage{amsfonts} and \mathbb{E}; not imply: \usepackage{amssymb} and \nRightarrow

  5. Encoding error: try adding \UseRawInputEncoding or \usepackage[utf8x]{inputenc} or \usepackage{newunicodechar} \newunicodechar{fi}{fi}

Posted on | In ,

Initialization

shape = [batch,height,width,channel]
constant:

1
2
3
4
tf.zeros(shape)
tf.constant(0.4, shape)
tf.random_norm(shape, stddev=1e-1)
tf.truncated_norm(shape, stddev=1e-1)

variable:

1
2
3
4
5
tf.get_variable(varname, shape=[5,5,64,192], initializer=XXX)
initializer=tf.zeros_initializer() # bias
initializer=tf.random_normal_initializer(0.0, 0.02) 
initializer=tf.truncated_normal_initializer(0.0, 0.02)
initializer=tf.contrib.layers.xavier_initializer() # weight

Empirically, for weight initialization, xavier is better than truncated_norm, better than random_normal.

Summary for tensorboard

1
2
3
4
tf.summary.scalar('total_loss', total_loss) #scalar, histogram, etc
self.merged = tf.summary.merge_all()
train_writer = tf.summary.FileWriter(summary_folder, model.sess.graph) #include the sess graph
train_writer.add_summary(summary, ibatch)

after running the code, open tensorboard tensorboard --logdir='./train'

Checkpoint

1
2
3
4
5
6
7
8
9
10
# save all the variables
saver = tf.train.Saver() 
# save partial variables 
saver = tf.train.Saver({"my_v2": v2})
# save and restore
saver.save(sess, "/tmp/model.ckpt")
saver.restore(sess, "/tmp/model.ckpt")
# restore partial variables
init_fn = slim.assign_from_checkpoint_fn(checkpoint_path, slim.get_variables_to_restore())
init_fn(sess)

inspect the checkpoint file

1
2
3
4
5
6
7
from tensorflow.python import pywrap_tensorflow  
checkpoint_path = os.path.join(model_dir, "model.ckpt")  
reader = pywrap_tensorflow.NewCheckpointReader(checkpoint_path)  
var_to_shape_map = reader.get_variable_to_shape_map()  
for key in var_to_shape_map:  
    print("tensor_name: ", key)  
    print(reader.get_tensor(key)) # Remove this is you want to print only variable names

Layer

  • convolutional layer

    1
    2
    3
    4
    kernel = tf.Variable(tf.zeros([5,5,64,192]))
    tf.nn.conv2d(x, kernel, strides=[1, 1, 1, 1], padding='SAME') # padding is 'SAME' or 'VALID' depending on stride is 1 or larger

    tf.nn.bias_add(prev_layer, bias)

  • dropout layer

    1
    2
    3
    keep_prob = tf.placeholder(tf.float32)
    # set keep_prob as 0.5 during training and 1 during testing
    tf.nn.dropout(h_fc1, keep_prob)

  • pooling layer

    1
    tf.nn.max_pool(x, ksize=[1, 2, 2, 1],  strides=[1, 2, 2, 1], padding='VALID')

  • loss layer

    1
    2
    tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y))
    	tf.nn.l2_loss(prev_layer)

  • metric layer

    1
    2
    # calculate top-k accuracy
    tf.nn.in_top_k(logits, label_holder, k) # the accuracy of top k
  • activation layer
    1
    tf.nn.relu(prev_layer)
  • other layers
    1
    2
    3
    tf.one_hot(self.input_class, self.n_class)
    tf.nn.embedding_lookup(codebook, indices)
    tf.nn.lrn() # legacy

Common Operations

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
tf.matmul
tf.pow
tf.diag_part
tf.reduce_sum
tf.reduce_mean
tf.nn.softmax
tf.tile(a,(2,2)) 
tf.nn.embedding_lookup
sim.flatten
tf.squeeze
tf.reshape
tf.concat(values=[a,b,c], axis=3)
tf.gather  #get indexed value, only on 0-dim
tf.gather_nd
tf.map_fn
tf.scan_fn
tf.fold1/foldr

Training and testing

  • training stage

    1
    2
    3
    4
    optimizer = tf.train.GradientDescentOptimizer(0.01)
    train = optimizer.minimize(loss)
    for i in range(1000): #note the iteration number here
        sess.run(train, feed_dict={x:x_train, y:y_train})

  • testing stage

    1
    2
    acc.eval(feed_dict={x:x_test, y:y_test})
    sess.run(acc, feed_dict={x:x_test, y:y_test}

Queue

  • For plain TensorFlow, reading data via queue is cumbersome. Refer to my another blog ‘TensorFlow Input Data’ for more details.
    ``python
    with tf.Session(config=config) as sess: 

      coord = tf.train.Coordinator()
  threads = tf.train.start_queue_runners(sess=sess, coord=coord)
  sess.run(tf.initialize_local_variables())
  #main code
  coord.request_stop()
  coord.join(threads)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    * With tfrecord and slim, using queue is much easier by using 	`slim.queues.QueueRunners`
    	```python
    	with tf.Session(config=config) as sess:        
    		with slim.queues.QueueRunners(sess):
    	``` 
      or `sv.managed_session`
    	```python
    	sv = tf.train.Supervisor(logdir=logdir, save_summaries_secs=0, saver=None)        
    	    with sv.managed_session() as sess:

    Sample code of generating tfrecord dataset and reading data via queue can downloaded here

Utils

  • check tensorflow version

    1
    tf.__version__.split('.')[0] != "1"

  • count parameter numbers

    1
    parameter_count = tf.reduce_sum([tf.reduce_prod(tf.shape(v)) for v in tf.trainable_variables()])

Posted on | In ,

  1. Input raw data (e.g., text, image address). There are many different implementations of queue.

  2. A more elegant way is converting raw data to tfrecord format.

Tips:

  1. setting large number_of_threading (e.g., 10) is helpful.

  2. shuffle the training samples to avoid homogenuity when necessary.

  3. place the training data in local disk instead of removable disk (consider I/O speed).

Posted on | In ,

All the GPU memory will be notoriously filled up even if you designate one GPU device.

maximum fraction

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.333)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))

automatic growth

config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)

visible GPU:

os.environ[“CUDA_VISIBLE_DEVICES”] = “0,1” # use python to set environment variables

use multiple GPUs

One typical to use mulitple GPU is to average gradients, please refer to the sample code.

Posted on | In ,

slim = tf.contrib.slim

Layers

for certain functions, assign default values to certain parameters

with slim.arg_scope([func1, func2, ....], arg1=val1, arg2=val2, ....)

https://www.tensorflow.org/api_docs/python/tf/contrib/layers

  • slim.conv2d
  • slim.max_pool2d
  • slim.avg_pool2d
  • slim.dropout
  • slim.batch_norm
  • slim.softmax
  • tf.repeat(inputs, repetitions, layer, args, *kwargs)

class Block(collections.namedtuple(‘Block’, [‘scope’, ‘unit_fn’, ‘args’])):
net = slim.utils.collect_named_outputs(outputs_collections, sc.name, net)

Load pretrained model

Note that initialization function must be associated with sess.

1
2
init_fn = slim.assign_from_checkpoint_fn(checkpoint_path, slim.get_variables_to_restore())
init_fn(sess)

Batch normalization

With slim:

1
2
3
4
5
6
7
8
9
batch_norm_params = {
	    # Decay for the moving averages.
	    'decay': batch_norm_decay,
	    # epsilon to prevent 0s in variance.
	    'epsilon': batch_norm_epsilon,
	    # collection containing update_ops.
	    'updates_collections': tf.GraphKeys.UPDATE_OPS,
 }
slim.arg_scope([slim.conv2d],  normalizer_fn=slim.batch_norm, normalizer_params=normalizer_params)

For tf.contrib.layers or tf.slim, when is_training=True, mean and variance based on each batch are used and moving_mean and moving_variance are updated if applicable. When is_training=False, loaded moving-mean an moving_variance are used.

To launch the update of moving_mean and moving_variance, special attention needs to be paid because this update operation is detached from gradient descent, which can be realized in the following ways.

The first method:

1
2
3
4
5
6
7
8
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
variables_to_train = _get_variables_to_train()
grads_and_vars = optimizer.compute_gradients(total_loss,  variables_to_train)
grad_updates = optimizer.apply_gradients(grads_and_vars)
update_ops.append(grad_updates)
update_op = tf.group(*update_ops)
with tf.control_dependencies([update_op]):
    train_op = tf.identity(total_loss)

The second method:

1
2
3
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
	train_op = optimizer.minimize(loss)

The third method:

1
train_op = slim.learning.create_train_op(total_loss, optimizer)

Otherwise, one can set updates_collections=None in slim.batch_norm to force the updates in place, but that can have a speed penalty, especially in distributed settings.

However, when trained on small-scale datasets, using moving_mean and moving_variance in the test stage often leads to extremely poor performance (close to random guess). This is due to the code start which renders moving_mean/variance unstable. There are two ways to fix the cold-start issue:

  • in the testing stage, also set is_training=True, i.e., use the mean and variance based on each test batch.

  • decrease batch_norm running average decay from default 0.999 to something like 0.99, which can speed up the start-up. When tuning decay, there is a trade-off between warm-up speed and statistical accuracy. For small-scale datasets, warm-up may take exceedingly long time, e.g., 300 epochs.

without slim: tf.nn.batch_normalization, no moving_mean/variance

1
2
3
4
5
6
7
8
9
10
11
def batchnorm(bn_input):
	with tf.variable_scope("batchnorm"):
		# this block looks like it has 3 inputs on the graph unless we do this
		bn_input = tf.identity(bn_input)
		
		channels = bn_input.get_shape()[3]
		offset = tf.get_variable("offset", [channels], dtype=tf.float32, initializer=tf.zeros_initializer())
		scale = tf.get_variable("scale", [channels], dtype=tf.float32, initializer=tf.random_normal_initializer(1.0, 0.02))
		mean, variance = tf.nn.moments(bn_input, axes=[0, 1, 2], keep_dims=False)
		normalized = tf.nn.batch_normalization(bn_input, mean, variance, offset, scale, variance_epsilon=1e-5)
		return normalized

Utils

print all model variables

1
2
3
4
5
tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES) # all the global variables
slim.get_model_variables() or tf.get_collection(tf.GraphKeys.MODEL_VARIABLES) 
#variables defined by slim (tf.contrib.framework.model_variable)
#excluding gradient variables
tf.trainable_variables() #excluding graident variables and batch_norm variables (moving_mean and moving_variance)

print regularization losses(weight decay) and other losses

1
2
3
slim.losses.get_regularization_losses()
slim.losses.get_losses() # losses except weight decay
slim.losses.get_total_loss(add_regularization_losses=False)

Posted on | In ,

Using multiple GPUs

device = torch.device(“cuda:gpu_id1”)

model = nn.DataParallel(model, [gpu_id1, gpu_id2, …])
model.to(device)

input = input.to(device)

Note that gpu_id1 must be the first gpu in the gpu_list in model.DataParallel(arg1, gpu_list)

Posted on | In ,

Framework Code

  1. Simplest App:

    • MyApp(False)means noredirect while MyApp(True, 'output.txt') will redirect the output to the file output.txt.

    • Derivation from wx.App uses def OnInit(self) while others use def __init__(self)

    • Donnot forget return True

  2. Simplest Frame: frame.Show(), frame.Centre()

  3. A frame with menubar, toolbar, and statusbar: UpdateUIEvents are sent periodically by the framework during idle time to allow the application to check if the state of a control needs to be updated.

    • Append a menu_item with icon, 

      qmi = wx.MenuItem(fileMenu, APP_EXIT, '&Quit\tCtrl+Q')
   qmi.SetBitmap(wx.Bitmap('exit.png'))
   fileMenu.AppendItem(qmi)

    • Menu check_item

      self.shtl = viewMenu.Append(wx.ID_ANY, 'Show toolbar', 'Show Toolbar', kind=wx.ITEM_CHECK)
   self.Bind(wx.EVT_MENU, self.ToggleStatusBar, self.shst)
   def ToggleStatusBar(self, e):        
       if self.shst.IsChecked():
        self.statusbar.Show()
       else:
           self.statusbar.Hide()

  4. Pop a Dialogue box with validator: derivation from wx.PyValidator

  5. A login dialogue before main frame: wx.PyValidator

  6. Notebook with multiple pages

  7. Advanced notebook, user can add new pages

  8. Foldable Pannel

  9. Splash Screen: splash before entering main frame wx.SplashScreen

  10. Extendable Pannel

  11. SplitPanel (hide, show): a horizontally or vertically splited panel, loading html file

  12. File Drag Drop

  13. File Hunter

  14. SpreadSheet

  15. Media Player

  16. Web Browser

Component Code

  1. Bitmap: use bitmap to beautify the appearance, or use the bitmap brush (transparent widgets may be a little troublesome here)

    self.Bind(wx.EVT_PAINT, self.OnPaint)
 def OnPaint(self, event):
     dc = wx.PaintDC(self)
     dc.SetBackgroundMode(wx.TRANSPARENT)
     brush1 = wx.BrushFromBitmap(wx.Bitmap('pattern1.jpg'))
     dc.SetBrush(brush1)
     w, h =  self.GetSize()
     dc.DrawRectangle(0, 0, w, h)

  2. Simplest Button: use lib to build advanced buttons

    • Bind function with building blocks 
      self.Bind(wx.EVT_BUTTON, self.OnButton, button) 
   def OnButton(self, event):
    • Note GetChildren(), GetParent(), GetId(), FindWindowById(self.btnId)

  3. Advanced button: bitmap button, toggle button, gradient button

  4. Frame Icon: personalize the frame icon

    con = wx.Icon(path, wx.BITMAP_TYPE_PNG)
 self.SetIcon(icon):

  5. Interaction with Clipboard: paste to and copy from system clipboard

  6. Drop Files to Frame: : wx.PyDropTarget

  7. Help in Frame: when initializing Frame

    pre = wx.PreFrame()
 pre.SetExtraStyle(wx.FRAME_EX_CONTEXTHELP)
 pre.Create(parent, *args, **kwargs)
 self.PostCreate(pre)

  8. Checkbox: use more general event detector to simplify code

    `self.Bind(wx.EVT_CHECKBOX, self.OnCheck)`

    e_obj = event.GetEventObject()

  9. dropdown menu

  10. MessageBox

    def ShowMessage(self,event):
     wx.MessageBox('Download completed', 'Info', wx.OK | wx.ICON_INFORMATION)

  11. Open file_dialogue

    dlg = wx.FileDialog(self, "Open File", style=wx.FD_OPEN)
 if dlg.ShowModal() == wx.ID_OK:
     fname = dlg.GetPath()
     handle = open(fname, 'r')
     self.txtctrl.SetValue(handle.read())
     handle.close()

  12. popup menu: right click to pop up the menu

  13. static box: a static box containing components

  14. list ctrl: multi-column list

  15. customtree: a tree-structure file browser

  16. virtual list box

  17. Styled Text: i.e., python-style text

  18. Download Progressbar

  19. About info: info including name, version, copyright, and description

  20. Choose Color Dialogue

    colour_data = wx.ColourData()
 colour = self.GetBackgroundColour()
 colour_data.SetColour(colour)
 colour_data.SetChooseFull(True)

 dlg = wx.ColourDialog(self, colour_data)
 if dlg.ShowModal() == wx.ID_OK:
     colour = dlg.GetColourData().GetColour()
     self.SetBackgroundColour(colour)
     self.Refresh()
 dlg.Destroy()

  21. Image Browser with simple editing

  22. Image Slide Show

  23. Search Bar

  24. Timer

    self._timer = wx.Timer(self)
 self.Bind(wx.EVT_TIMER, self.OnTimer, self._timer)
 self._timer.Start(100)
 self._timer.Stop()

  25. Execute Command Line

    import outputwin
 self.output = outputwin.OutputWindow(self)
 self.output.StartProcess("ping %s" % url, blocksize=64)

  26. Music Player

  27. Video Player: First, you need to install MplayerCtrl lib. Secondly, place the mplayer folder under the current working directory.

Layout

  1. wx.BoxSizer: proportion is used to control main direction and wx.EXPAND is used to control the other direction. Note in BoxSizer, alignment is only valid in one direction. AddSpacer(50) is equal to Add((50,50)). AddStretchSpacer() is equal to Add((0,0),proportion=1).

    sizer = wx.BoxSizer(wx.HORIZONTAL)
 sizer.AddSpacer(50)
 sizer.Add(sth,proportion=0, flag=wx.ALL, border=5) #use flag to mark which side has border
 sizer.Add((-1,10)) #add a black space, height=10
 # sizer.Add(sth,proportion=0, wx.EXPAND|wx.RIGHT|wx.ALIGN_RIGHT, border=5)
 sizer.AddSpacer((0,0)) #sizer.AddStretchSpacer()
 self.SetSizer(sizer)
 self.SetInitialSize()

  2. wx.GridSizer: proportion is usually set as 0, use Add((20,20), 1, wx.EXPAND) to take up space.

    wx.GridSizer(2, 2, vgap=0, hgap=0)
 msizer.Add(sth, 0, wx.EXPAND)

  3. wx.FlexGridSizer: make some rows and columns growable.

    fgs.AddGrowableRow(2)
 fgs.AddGrowableCol(1)

  4. wx.GridBagSizer: use pos and span to indicate the location and size.

    sizer = wx.GridBagSizer(vgap=8, hgap=8)
 sizer.Add(sth, (1, 2), (1, 15), wx.EXPAND)

Notes

  1. Event Propagation: When an event can intrigue multiple events, use event.skip() to guaranttee the occurrence of following events. Take keyevents.py for an example.

  2. Virtual Ride: wx.PyPannel

  3. Bind function which will be checked in the idle time
    self.Bind(wx.EVT_UPDATE_UI, self.OnUpdateEditMenu)

Posted on | In ,

  1. check type

    1
    isinstance(n,int)

  2. list

    1
    2
    3
    4
    5
    6
    7
    8
    t1 = ['a', 'b', 'c'];
    t2 = ['d', 'e', 'f'];
    t1.extend(t2);
    t1.append('g');
    t1 = t1+['g']
    t1.sort();
    del t1[1:3];
    t1.remove('b');

  3. dictionary

    1
    2
    3
    4
    5
    6
    7
    eng2sp = {'one': 'uno', 'two': 'dos', 'three': 'tres'}
    eng2sp.values() #'uno','dos','tres'
    eng2sp.items()
    'one' in eng2sp #true
    inverse = invert_dict(eng2sp)  
    d = dict(zip('abc',range(3)))
    d.get(word,0) #d[word] if word in d, 0 otherwise

  4. tupe: similar with list but immutable

  5. file operation

    1
    2
    3
    4
    5
    6
    7
    8
    fout = os.getcwd()
    os.path.abspath('tmp.txt')
    os.path.exists('tmp.txt')
    os.path.isfile/isdir
    os.listdir(cwd)
    fout = open('output.txt','w')
    fout.write('hehe')
    fout.close()

Posted on | In ,

  1. Open SMTP service for your email and obtain the SMTP password.

  2. Create a Python script

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    import win32serviceutil
    import win32service
    import win32event
    import servicemanager
    import smtplib
    import time
    import re
    import sys
    import urllib2

    class Getmyip:
        def visit(self,url):
            opener = urllib2.urlopen(url)
            if url == opener.geturl():
                IPstr = opener.read()
            return re.search('\d+\.\d+\.\d+\.\d+',IPstr).group(0)
     
        def getip(self):
            myip = self.visit("http://www.net.cn/static/customercare/yourip.asp")
            return myip
        
    class AppServerSvc (win32serviceutil.ServiceFramework):
        _svc_name_ = "TestService"
        _svc_display_name_ = "Test Service"

        def __init__(self,args):
            win32serviceutil.ServiceFramework.__init__(self,args)
            self.hWaitStop = win32event.CreateEvent(None,0,0,None)
            self.run = True

        def SvcStop(self):
            self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
            win32event.SetEvent(self.hWaitStop)
            self.run = False
            
        def SvcDoRun(self):
            while(self.run==True):
                self.main()

        def main(self):
            fromaddr = 'XXX@qq.com'
            toaddrs  = 'XXX@qq.com' 
            
            server = smtplib.SMTP_SSL('smtp.qq.com')
            server.set_debuglevel(1)    
            print("--- Need Authentication ---")
            username = 'XXX' 
            password = 'XXX' 
            server.login(username, password)
            
            prev_msg = ''
            
            while(True):
                getmyip=Getmyip()
                msg = getmyip.getip()
                if not msg==prev_msg:
                    server.sendmail(fromaddr, toaddrs, msg)
                    prev_msg = msg
                time.sleep(10)
            
            server.quit()

    if __name__ == '__main__':
        if len(sys.argv) == 1:
            servicemanager.Initialize()
            servicemanager.PrepareToHostSingle(AppServerSvc)
            servicemanager.StartServiceCtrlDispatcher()
        else:
            win32serviceutil.HandleCommandLine(AppServerSvc)

  3. Make python file into an exe

    1
    pyinstaller -F MyService.py

  4. Make exe into a Windows service

    1
    2
    3
    4
    sc create MyServer binPath=$exe_path
    sc start MyServer
    sc stop MyServer
    sc delete MyServer

Posted on | In ,

Use PIL

1
2
3
4
5
6
7
8
import PIL.ImageDraw
import PIL.Image
import PIL.ImageFont

pil_img = PIL.Image.fromarray(bb_img)
draw = PIL.ImageDraw.Draw(pil_img)
font = PIL.ImageFont.truetype("/usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf",20)
draw.text((10, 10),"Frame: %09d"%(iframe),(255,0,0),font=font)

Use OpenCV

—

1
2
import cv2
cv2.putText(cv_img, 'Frame: %d'%(iframe), (10,20),  cv2.FONT_HERSHEY_SIMPLEX, 0.65, (0,0,255), 3)