Newly Blog

• tutorial: [1]

1. https://github.com/yuki-koyama/blender-cli-rendering
2. https://github.com/FrozenBurning/Text2Light

Initialization

shape = [batch,height,width,channel]
constant:

1
2
3
4

tf.zeros(shape)
tf.constant(0.4, shape)
tf.random_norm(shape, stddev=1e-1)
tf.truncated_norm(shape, stddev=1e-1)

1
2
3
4
5

tf.get_variable(varname, shape=[5,5,64,192], initializer=XXX)
initializer=tf.zeros_initializer() # bias
initializer=tf.random_normal_initializer(0.0, 0.02) 
initializer=tf.truncated_normal_initializer(0.0, 0.02)
initializer=tf.contrib.layers.xavier_initializer() # weight

Summary for tensorboard

1
2
3
4

tf.summary.scalar('total_loss', total_loss) #scalar, histogram, etc
self.merged = tf.summary.merge_all()
train_writer = tf.summary.FileWriter(summary_folder, model.sess.graph) #include the sess graph
train_writer.add_summary(summary, ibatch)

after running the code, open tensorboard tensorboard --logdir='./train'

Checkpoint

1
2
3
4
5
6
7
8
9
10

# save all the variables
saver = tf.train.Saver() 
# save partial variables 
saver = tf.train.Saver({"my_v2": v2})
# save and restore
saver.save(sess, "/tmp/model.ckpt")
saver.restore(sess, "/tmp/model.ckpt")
# restore partial variables
init_fn = slim.assign_from_checkpoint_fn(checkpoint_path, slim.get_variables_to_restore())
init_fn(sess)

inspect the checkpoint file

1
2
3
4
5
6
7

from tensorflow.python import pywrap_tensorflow  
checkpoint_path = os.path.join(model_dir, "model.ckpt")  
reader = pywrap_tensorflow.NewCheckpointReader(checkpoint_path)  
var_to_shape_map = reader.get_variable_to_shape_map()  
for key in var_to_shape_map:  
    print("tensor_name: ", key)  
    print(reader.get_tensor(key)) # Remove this is you want to print only variable names  

Layer

• convolutional layer

  1 2 3 4

  kernel = tf.Variable(tf.zeros([5,5,64,192])) tf.nn.conv2d(x, kernel, strides=[1, 1, 1, 1], padding='SAME') # padding is 'SAME' or 'VALID' depending on stride is 1 or larger tf.nn.bias_add(prev_layer, bias)

• dropout layer

  1 2 3

  keep_prob = tf.placeholder(tf.float32) # set keep_prob as 0.5 during training and 1 during testing tf.nn.dropout(h_fc1, keep_prob)

• pooling layer

  1	tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

• loss layer

  1 2	tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y)) tf.nn.l2_loss(prev_layer)

• metric layer

  1 2	# calculate top-k accuracy tf.nn.in_top_k(logits, label_holder, k) # the accuracy of top k

• activation layer

  1	tf.nn.relu(prev_layer)

• other layers

  1 2 3

  tf.one_hot(self.input_class, self.n_class) tf.nn.embedding_lookup(codebook, indices) tf.nn.lrn() # legacy

Common Operations

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

tf.matmul
tf.pow
tf.diag_part
tf.reduce_sum
tf.reduce_mean
tf.nn.softmax
tf.tile(a,(2,2)) 
tf.nn.embedding_lookup
sim.flatten
tf.squeeze
tf.reshape
tf.concat(values=[a,b,c], axis=3)
tf.gather  #get indexed value, only on 0-dim
tf.gather_nd
tf.map_fn
tf.scan_fn
tf.fold1/foldr 

Training and testing

• training stage

  1 2 3 4

  optimizer = tf.train.GradientDescentOptimizer(0.01) train = optimizer.minimize(loss) for i in range(1000): #note the iteration number here sess.run(train, feed_dict={x:x_train, y:y_train})

• testing stage

  1 2	acc.eval(feed_dict={x:x_test, y:y_test}) sess.run(acc, feed_dict={x:x_test, y:y_test}

Queue

• For plain TensorFlow, reading data via queue is cumbersome. Refer to my another blog ‘TensorFlow Input Data’ for more details.
  ``python
  with tf.Session(config=config) as sess:

    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(sess=sess, coord=coord)
    sess.run(tf.initialize_local_variables())
    #main code
    coord.request_stop()
    coord.join(threads)
  

  1 2 3 4

  * With tfrecord and slim, using queue is much easier by using `slim.queues.QueueRunners` ```python with tf.Session(config=config) as sess: with slim.queues.QueueRunners(sess):

  or sv.managed_session

  1 2	sv = tf.train.Supervisor(logdir=logdir, save_summaries_secs=0, saver=None) with sv.managed_session() as sess:

  Sample code of generating tfrecord dataset and reading data via queue can downloaded here

Utils

• check tensorflow version

  1	tf.__version__.split('.')[0] != "1"

• count parameter numbers

  1	parameter_count = tf.reduce_sum([tf.reduce_prod(tf.shape(v)) for v in tf.trainable_variables()])

1. Input raw data (e.g., text, image address). There are many different implementations of queue.

   • feature-label pair: [enqueue-op] [fetch-func] [slice-input-producer] [string-input-producer]

   • image-image pair: [slice-input-producer][string-input-producer]

2. A more elegant way is converting raw data to tfrecord format.

Tips:

1. setting large number_of_threading (e.g., 10) is helpful.

2. shuffle the training samples to avoid homogenuity when necessary.

3. place the training data in local disk instead of removable disk (consider I/O speed).

All the GPU memory will be notoriously filled up even if you designate one GPU device.

maximum fraction

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.333)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))

automatic growth

config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)

visible GPU:

os.environ[“CUDA_VISIBLE_DEVICES”] = “0,1” # use python to set environment variables

use multiple GPUs

One typical to use mulitple GPU is to average gradients, please refer to the sample code.

slim = tf.contrib.slim

Layers

for certain functions, assign default values to certain parameters

with slim.arg_scope([func1, func2, ....], arg1=val1, arg2=val2, ....)

https://www.tensorflow.org/api_docs/python/tf/contrib/layers

• slim.conv2d
• slim.max_pool2d
• slim.avg_pool2d
• slim.dropout
• slim.batch_norm
• slim.softmax
• tf.repeat(inputs, repetitions, layer, args, *kwargs)

class Block(collections.namedtuple(‘Block’, [‘scope’, ‘unit_fn’, ‘args’])):
net = slim.utils.collect_named_outputs(outputs_collections, sc.name, net)

Load pretrained model

Note that initialization function must be associated with sess.

1
2

init_fn = slim.assign_from_checkpoint_fn(checkpoint_path, slim.get_variables_to_restore())
init_fn(sess)

Batch normalization

With slim:

1
2
3
4
5
6
7
8
9

batch_norm_params = {
	    # Decay for the moving averages.
	    'decay': batch_norm_decay,
	    # epsilon to prevent 0s in variance.
	    'epsilon': batch_norm_epsilon,
	    # collection containing update_ops.
	    'updates_collections': tf.GraphKeys.UPDATE_OPS,
 }
slim.arg_scope([slim.conv2d],  normalizer_fn=slim.batch_norm, normalizer_params=normalizer_params) 

For tf.contrib.layers or tf.slim, when is_training=True, mean and variance based on each batch are used and moving_mean and moving_variance are updated if applicable. When is_training=False, loaded moving-mean an moving_variance are used.

To launch the update of moving_mean and moving_variance, special attention needs to be paid because this update operation is detached from gradient descent, which can be realized in the following ways.

The first method:

1
2
3
4
5
6
7
8

update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
variables_to_train = _get_variables_to_train()
grads_and_vars = optimizer.compute_gradients(total_loss,  variables_to_train)
grad_updates = optimizer.apply_gradients(grads_and_vars)
update_ops.append(grad_updates)
update_op = tf.group(*update_ops)
with tf.control_dependencies([update_op]):
    train_op = tf.identity(total_loss) 

The second method:

1
2
3

update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
	train_op = optimizer.minimize(loss)

The third method:

1

train_op = slim.learning.create_train_op(total_loss, optimizer)

Otherwise, one can set updates_collections=None in slim.batch_norm to force the updates in place, but that can have a speed penalty, especially in distributed settings.

However, when trained on small-scale datasets, using moving_mean and moving_variance in the test stage often leads to extremely poor performance (close to random guess). This is due to the code start which renders moving_mean/variance unstable. There are two ways to fix the cold-start issue:

• in the testing stage, also set is_training=True, i.e., use the mean and variance based on each test batch.

• decrease batch_norm running average decay from default 0.999 to something like 0.99, which can speed up the start-up. When tuning decay, there is a trade-off between warm-up speed and statistical accuracy. For small-scale datasets, warm-up may take exceedingly long time, e.g., 300 epochs.

without slim: tf.nn.batch_normalization, no moving_mean/variance

1
2
3
4
5
6
7
8
9
10
11

def batchnorm(bn_input):
	with tf.variable_scope("batchnorm"):
		# this block looks like it has 3 inputs on the graph unless we do this
		bn_input = tf.identity(bn_input)
		
		channels = bn_input.get_shape()[3]
		offset = tf.get_variable("offset", [channels], dtype=tf.float32, initializer=tf.zeros_initializer())
		scale = tf.get_variable("scale", [channels], dtype=tf.float32, initializer=tf.random_normal_initializer(1.0, 0.02))
		mean, variance = tf.nn.moments(bn_input, axes=[0, 1, 2], keep_dims=False)
		normalized = tf.nn.batch_normalization(bn_input, mean, variance, offset, scale, variance_epsilon=1e-5)
		return normalized	

Utils

print all model variables

1
2
3
4
5

tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES) # all the global variables
slim.get_model_variables() or tf.get_collection(tf.GraphKeys.MODEL_VARIABLES) 
#variables defined by slim (tf.contrib.framework.model_variable)
#excluding gradient variables
tf.trainable_variables() #excluding graident variables and batch_norm variables (moving_mean and moving_variance)

1
2
3

slim.losses.get_regularization_losses()
slim.losses.get_losses() # losses except weight decay
slim.losses.get_total_loss(add_regularization_losses=False)

Using multiple GPUs

device = torch.device(“cuda:gpu_id1”)

model = nn.DataParallel(model, [gpu_id1, gpu_id2, …])
model.to(device)

input = input.to(device)

Note that gpu_id1 must be the first gpu in the gpu_list in model.DataParallel(arg1, gpu_list)

Framework Code

1. Simplest App:

   • MyApp(False)means noredirect while MyApp(True, 'output.txt') will redirect the output to the file output.txt.

   • Derivation from wx.App uses def OnInit(self) while others use def __init__(self)

   • Donnot forget return True

2. Simplest Frame: frame.Show(), frame.Centre()

3. A frame with menubar, toolbar, and statusbar: UpdateUIEvents are sent periodically by the framework during idle time to allow the application to check if the state of a control needs to be updated.

   • Append a menu_item with icon,

     qmi = wx.MenuItem(fileMenu, APP_EXIT, '&Quit\tCtrl+Q')
        qmi.SetBitmap(wx.Bitmap('exit.png'))
        fileMenu.AppendItem(qmi)

   • Menu check_item

     self.shtl = viewMenu.Append(wx.ID_ANY, 'Show toolbar', 'Show Toolbar', kind=wx.ITEM_CHECK)
        self.Bind(wx.EVT_MENU, self.ToggleStatusBar, self.shst)
        def ToggleStatusBar(self, e):        
            if self.shst.IsChecked():
             self.statusbar.Show()
            else:
                self.statusbar.Hide()

4. Pop a Dialogue box with validator: derivation from wx.PyValidator

5. A login dialogue before main frame: wx.PyValidator

6. Notebook with multiple pages

7. Advanced notebook, user can add new pages

8. Foldable Pannel

   

9. Splash Screen: splash before entering main frame wx.SplashScreen

10. Extendable Pannel

    

11. SplitPanel (hide, show): a horizontally or vertically splited panel, loading html file

12. File Drag Drop

    

13. File Hunter

    

14. SpreadSheet

    

15. Media Player

    

16. Web Browser

    

Component Code

1. Bitmap: use bitmap to beautify the appearance, or use the bitmap brush (transparent widgets may be a little troublesome here)

   self.Bind(wx.EVT_PAINT, self.OnPaint)
    def OnPaint(self, event):
        dc = wx.PaintDC(self)
        dc.SetBackgroundMode(wx.TRANSPARENT)
        brush1 = wx.BrushFromBitmap(wx.Bitmap('pattern1.jpg'))
        dc.SetBrush(brush1)
        w, h =  self.GetSize()
        dc.DrawRectangle(0, 0, w, h)

2. Simplest Button: use lib to build advanced buttons

   • Bind function with building blocks

     self.Bind(wx.EVT_BUTTON, self.OnButton, button) 
        def OnButton(self, event):

   • Note GetChildren(), GetParent(), GetId(), FindWindowById(self.btnId)

3. Advanced button: bitmap button, toggle button, gradient button

4. Frame Icon: personalize the frame icon

   con = wx.Icon(path, wx.BITMAP_TYPE_PNG)
    self.SetIcon(icon):

5. Interaction with Clipboard: paste to and copy from system clipboard

6. Drop Files to Frame: : wx.PyDropTarget

7. Help in Frame: when initializing Frame

   pre = wx.PreFrame()
    pre.SetExtraStyle(wx.FRAME_EX_CONTEXTHELP)
    pre.Create(parent, *args, **kwargs)
    self.PostCreate(pre)

8. Checkbox: use more general event detector to simplify code

   `self.Bind(wx.EVT_CHECKBOX, self.OnCheck)`
   

   e_obj = event.GetEventObject()

9. dropdown menu

10. MessageBox

    def ShowMessage(self,event):
         wx.MessageBox('Download completed', 'Info', wx.OK | wx.ICON_INFORMATION)

11. Open file_dialogue

    dlg = wx.FileDialog(self, "Open File", style=wx.FD_OPEN)
     if dlg.ShowModal() == wx.ID_OK:
         fname = dlg.GetPath()
         handle = open(fname, 'r')
         self.txtctrl.SetValue(handle.read())
         handle.close()

12. popup menu: right click to pop up the menu

13. static box: a static box containing components

14. list ctrl: multi-column list

    

15. customtree: a tree-structure file browser

16. virtual list box

    

17. Styled Text: i.e., python-style text

18. Download Progressbar

19. About info: info including name, version, copyright, and description

20. Choose Color Dialogue

    colour_data = wx.ColourData()
     colour = self.GetBackgroundColour()
     colour_data.SetColour(colour)
     colour_data.SetChooseFull(True)
    
     dlg = wx.ColourDialog(self, colour_data)
     if dlg.ShowModal() == wx.ID_OK:
         colour = dlg.GetColourData().GetColour()
         self.SetBackgroundColour(colour)
         self.Refresh()
     dlg.Destroy()

21. Image Browser with simple editing

22. Image Slide Show

23. Search Bar

24. Timer

    self._timer = wx.Timer(self)
     self.Bind(wx.EVT_TIMER, self.OnTimer, self._timer)
     self._timer.Start(100)
     self._timer.Stop()

25. Execute Command Line

    import outputwin
     self.output = outputwin.OutputWindow(self)
     self.output.StartProcess("ping %s" % url, blocksize=64)

26. Music Player

27. Video Player: First, you need to install MplayerCtrl lib. Secondly, place the mplayer folder under the current working directory.

Layout

1. wx.BoxSizer: proportion is used to control main direction and wx.EXPAND is used to control the other direction. Note in BoxSizer, alignment is only valid in one direction. AddSpacer(50) is equal to Add((50,50)). AddStretchSpacer() is equal to Add((0,0),proportion=1).

   sizer = wx.BoxSizer(wx.HORIZONTAL)
    sizer.AddSpacer(50)
    sizer.Add(sth,proportion=0, flag=wx.ALL, border=5) #use flag to mark which side has border
    sizer.Add((-1,10)) #add a black space, height=10
    # sizer.Add(sth,proportion=0, wx.EXPAND|wx.RIGHT|wx.ALIGN_RIGHT, border=5)
    sizer.AddSpacer((0,0)) #sizer.AddStretchSpacer()
    self.SetSizer(sizer)
    self.SetInitialSize()

2. wx.GridSizer: proportion is usually set as 0, use Add((20,20), 1, wx.EXPAND) to take up space.

   wx.GridSizer(2, 2, vgap=0, hgap=0)
    msizer.Add(sth, 0, wx.EXPAND)

3. wx.FlexGridSizer: make some rows and columns growable.

   fgs.AddGrowableRow(2)
    fgs.AddGrowableCol(1)

4. wx.GridBagSizer: use pos and span to indicate the location and size.

   sizer = wx.GridBagSizer(vgap=8, hgap=8)
    sizer.Add(sth, (1, 2), (1, 15), wx.EXPAND) 

Notes

1. Event Propagation: When an event can intrigue multiple events, use event.skip() to guaranttee the occurrence of following events. Take keyevents.py for an example.

2. Virtual Ride: wx.PyPannel

3. Bind function which will be checked in the idle time
   self.Bind(wx.EVT_UPDATE_UI, self.OnUpdateEditMenu)

1. check type

   1	isinstance(n,int)

2. list

   1 2 3 4 5 6 7 8

   t1 = ['a', 'b', 'c']; t2 = ['d', 'e', 'f']; t1.extend(t2); t1.append('g'); t1 = t1+['g'] t1.sort(); del t1[1:3]; t1.remove('b');

3. dictionary

   1 2 3 4 5 6 7

   eng2sp = {'one': 'uno', 'two': 'dos', 'three': 'tres'} eng2sp.values() #'uno','dos','tres' eng2sp.items() 'one' in eng2sp #true inverse = invert_dict(eng2sp) d = dict(zip('abc',range(3))) d.get(word,0) #d[word] if word in d, 0 otherwise

4. tupe: similar with list but immutable

5. file operation

   1 2 3 4 5 6 7 8

   fout = os.getcwd() os.path.abspath('tmp.txt') os.path.exists('tmp.txt') os.path.isfile/isdir os.listdir(cwd) fout = open('output.txt','w') fout.write('hehe') fout.close()

1. Open SMTP service for your email and obtain the SMTP password.

2. Create a Python script

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70

   import win32serviceutil import win32service import win32event import servicemanager import smtplib import time import re import sys import urllib2 class Getmyip: def visit(self,url): opener = urllib2.urlopen(url) if url == opener.geturl(): IPstr = opener.read() return re.search('\d+\.\d+\.\d+\.\d+',IPstr).group(0) def getip(self): myip = self.visit("http://www.net.cn/static/customercare/yourip.asp") return myip class AppServerSvc (win32serviceutil.ServiceFramework): _svc_name_ = "TestService" _svc_display_name_ = "Test Service" def __init__(self,args): win32serviceutil.ServiceFramework.__init__(self,args) self.hWaitStop = win32event.CreateEvent(None,0,0,None) self.run = True def SvcStop(self): self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING) win32event.SetEvent(self.hWaitStop) self.run = False def SvcDoRun(self): while(self.run==True): self.main() def main(self): fromaddr = 'XXX@qq.com' toaddrs = 'XXX@qq.com' server = smtplib.SMTP_SSL('smtp.qq.com') server.set_debuglevel(1) print("--- Need Authentication ---") username = 'XXX' password = 'XXX' server.login(username, password) prev_msg = '' while(True): getmyip=Getmyip() msg = getmyip.getip() if not msg==prev_msg: server.sendmail(fromaddr, toaddrs, msg) prev_msg = msg time.sleep(10) server.quit() if __name__ == '__main__': if len(sys.argv) == 1: servicemanager.Initialize() servicemanager.PrepareToHostSingle(AppServerSvc) servicemanager.StartServiceCtrlDispatcher() else: win32serviceutil.HandleCommandLine(AppServerSvc)

3. Make python file into an exe

   1	pyinstaller -F MyService.py

4. Make exe into a Windows service

   1 2 3 4

   sc create MyServer binPath=$exe_path sc start MyServer sc stop MyServer sc delete MyServer