Strong Induction and run-time complexity for recursive functions

        Hi, it’s nice to be back. In my previous slog, I mentioned about analyzing the run-time complexity of recursive function and the master theorem. So I will spend most of the time on it today by using merge sort as an example.

        At the very beginning, I would like to introduce a digression here – The effectiveness of strong induction in writing recursive function. The idea of strong induction is by having a functional base case and assuming the function works for all the smaller cases, the function will eventually solve a bigger problem. Like the inorder traversal we did in Lab9, I simply put

So I have a functional base case -  if a binary tree does not have left and right sub-tree, simply return the linked list with the root of the tree. Then there are three difference cases – A tree without left sub-tree, a tree without right sub-tree, and a complete tree. For the first two different cases, by strong induction, I assume my function works for smaller cases so I just changed where I prepend the root of the tree and simply call the function on sub-trees. Then for the last case, also by strong induction, I called the function on right sub-tree, prepend the root, then prepend the linked-list generated by calling ‘inorder’ on the left sub-tree. And then I’m done. By assuming the function works for smaller cases, I eventually solved a larger case. Note you need to change the prepend method a bit for prepending linked-lists. So through strong induction, all I need to do is to find out the base case and use smaller cases to construct a larger case and it will provide much ease when writing recursive function.

    It’s a huge digression actually ha-ha! Now I’m back on track. I’ll take the merge sort we did in class as an example.

Notice this is not the complete version since I don’t include merge function here due to space limit. So for merge sort, by strong induction, this should work as it has a valid base case and uses smaller cases to construct a larger case. But its run-time complexity is not easy to find out. Since the run-time complexity depends only on worst cases, I’ll ignore the condition when length of L is smaller than 2 but analyze the else part. The very first thing we need to do is to assume the total steps merge sort takes are M(n) steps for a list of length n. Then by our assumption, both merge_sort(L[:len(L)//2]) and merge_sort(L[len(L)//2:]) will take approximately M(n/2) steps by ignoring the constants because of the nature of big-Oh. For merge, since it’s a procedural function, we can easily analyze its total steps for merging two lists of n/2 length is c*n where c is a constant. So

M(n) = M(n/2) + M(n/2) + cn = 2M(n/2) + cn = 2(2M(n/4)+cn/2) + cn/4 = …. So by telescoping this equation , 

we can see M(n)/n  = logn eventually, so M(n) ∈ o(nlogn). For more information on proving this, I suggest you check this powerpoint ( http://www.cs.princeton.edu/courses/archive/spr07/cos226/lectures/04MergeQuick.pdf ). So the idea is to firstly find the recurrence for run-time complexity and then telescoping it. Eventually the recurrence could be solve and the big-Oh notation will come through.

    That’s it and good luck on term test2.

Introduction to Runtime Complexity

   Hey, sorry that this slog is a bit late since I was quite sick last week but now since I’ve come back, I can write down some of my own reflective on run-time complexity.

               First of all, what is the run-time complexity for a program? Basically it’s the total steps a program would take given the size of input n, and we formally use O(f(n)) to describe the asymptotic behavior of the program and here O(f(n)) is defined as ‘There exists a natural number c, and there exists a natural number b, such that for all n bigger than b, the total steps of the program would be less than or equal to c*f(n)’. Quite obscure and doesn’t seem to make sense at the first glance so here I am going to introduce my own interpretation: For a run-time complexity of a program to be in O(f(n)), the growth rate of the total steps of the program with respect to the size of input is less than or equal to f(n). So if the total steps a program takes is n^2 given the size of input n, then we will say the run-time complexity of the program belongs to O(n^2).

               Next, why should we even care about the run-time complexity of a program? Some may claim that as long as the program works, it is a good program. This makes sense when the input is relatively small but suppose here we are given the size 1 million, and one program has a run-time complexity of log(n) while the other has n^2. Log of one million is about 20 while the square of one million is one times ten to the 12^th and this is a huge difference since computers nowadays, though fast, still have limits and it’s obvious 20 steps takes much less time. So correctness differentiate bad and good programs while efficiency differentiate good and better programs.

               As I have told about the interpretation of run-time complexity and the reason why we should care about run-time complexity. I would like to go a bit further by telling you how to analyze the run-time complexity of a program. There’s a small trick I found: The depth of a for loop depends on the input size usually determines the power of n in O(n^x). Take the following program as an example.

This program has a nested for loop so its depth is 2 and since both for loop depends on the size of input x. Thus the inner loop runs x times and the outer loop runs the inner loop x times and the run-time complexity of this program is, obviously O(n^2). Also, always remember that constants normally would be ignored when calculating the run-time complexity because of the definition of O(f(n)).

              Though my trick works for many programs, it has a limitation that such trick could only be applied in procedural programs while for recursion, more work needs to be done. I was stuck on how to calculate the run-time complexity for recursive programs until this week in CSC240’s lecture. I was told the master theorem (check http://en.wikipedia.org/wiki/Master_theorem for more info.) and then I have the weapon to tackle the run-time complexity of recursive programs. I’m still on the way of learning and I’ll come back later this week to write more on analyzing recursive programs.

What is linked list and when to use linked list

        During the last two week, we spent most of the time dealing with linked lists and I am always wondering why I should use linked lists. Now that I finally have a week without midterms, I can write down some thoughts.

First of all, what is a linked list? In my own viewpoint, linked list contains elements with two inner attributes and the first attribute points to an object while the second attribute referenced the element behind it. This might be a bit abstract but if you look the following graph, I am sure you can easily grasp the idea of a linked list.

        Like what I just said before, the first attribute points to a value while the second attribute serves as a pointer pointing to the next element so if you want to get the second element of the linked list, you need to follow the references from the first element to reach the second element and same for the third element, and the fourth element, etc. This idea of extracting the element from the previous element is the idea of recursion where the solution to one task depends on the solution to the previous task and the code is as follows. (Check http://siyangcsc148w3.blogspot.ca/2014/02/what-is-recursion.html for more on recursion if interested)

        However, when I was doing the lab on week 6, I found there is a serious problem associated with this way of getting elements. As I stated before, most languages have a maximum recursion limit to prevent the program from getting stuck in infinite loops, so suppose we have a linked list of 10,000 elements and we want to know the 5,000-th element, then due to the recursion limit, we cannot get the 5000-th element since it requires 4999 recursion which is far beyond the maximum stack limit which inevitably resulted in an error. So the only way to access an element is through step by step, find the previous element, then to the next element and this might seem quite inefficient compared with lists where we can just use the index of an element to reach that element. Thus someone may say, there is no need to have linked list and list is enough. I will, under most cases, agree with that statement but there are some cases where linked list is much more efficient since linked list allows for constant-time(O(1)) insertion and deletion of an element since during insertion of one element, all we did is to get to that position and change the pointer of the previous element to the element we want to insert and set the pointer of the newly added element to the original element in that position while not affecting other element. However, if we want to insert an element to the middle of a list, we had to move every element after it to the right to leave space for inserting the element we wanted.  As the length of lists increased, the time it takes to move all elements to the right gets longer and longer which is, obviously, quite inefficient. So for random elements, yes, we should use list since it allows for random indexing but if for sorted elements where we know exactly which place to insert an element, we definitely should use linked list. This idea is applied in binary-tree search in which a tree is built with nodes placed in order, and we could easily insert a node, changing the pointers or access that element from the root under the order we designed without concerning too much about the length of the linked list.

To sum up, when we want to place different object without any order, a list is a better approach to store these objects since list allows for random indexing, but if we want to place objects in an order, then linked list is a better approach since it allows for constant-time insertion and deletion.