next up previous


CSE 2320 Section 501/571 Fall 1999

Sample Dynamic Programming / Greedy Algorithms Solution

1.
Consider the problem of constructing an optimal binary search tree (in terms of search time) for a set of integer keys, given the frequencies with which each key will be accessed. Assume there are n keys with values 1 to n, and key i has frequency fi such that $0 \leq f_i \leq
1$ and $\sum_{i=1}^{n} f_i = 1$. The goal is to minimize the value of $\sum_{i=1}^{n} f_i d_i $ where di is the depth of key i in the binary search tree.

(a)
Let c(i,j) be the optimal cost of a binary search tree containing the keys $i \ldots j$. Assuming the optimal choice for the root of this tree is key k, give an expression for c(i,j) in terms of c(i,k-1) and c(k+1,j).


\begin{displaymath}c(i,j) = c(i,k-1) + c(k+1,j) + \left( \sum_{l=i}^{j} f_l \right) - f_k\end{displaymath}

The last two terms derive from the fact that attaching the left and right subtrees to the root key k adds one to the depth of each key in the subtrees. Thus, each fi di term in the original summation becomes fi(di + 1), which adds an extra fi to the cost for each key in the subtree, except for key k.

(b)
Show that this problem exhibits optimal substructure.

Let $c_{T}(i,j) = \sum_{l=i}^{j} f_l d_l$ be the cost of a binary search tree T for the keys i through j. Assume we have an optimal tree Topt for the keys i through j consisting of a root key k with left subtree TL containing keys i to k-1 and right subtree TRcontaining keys k+1 to j (see figure below).

\psfig{figure=figures/dpga1.ps}

We can express the cost of the tree Topt as


 \begin{displaymath}c_{T_{opt}}(i,j) = c_{T_L}(i,k-1) + c_{T_R}(k+1,j) +
\left( \sum_{l=i}^{j} f_l \right) - f_k
\end{displaymath} (1)

To prove optimal substructure, we need to show that the subtrees TL and TR contained in Topt are optimal binary search trees for the keys i to k-1 and k+1 to j, respectively. Using proof by contradiction, assume that there is a tree $T_L^{\prime}$ having a lower cost than TL; i.e., $c_{T_L^{\prime}}(i,k-1) < c_{T_L}(i,k-1)$. Then, we can build a new tree $T^{\prime}$ by replacing TL by $T_L^{\prime}$ in Topt. The cost of tree $T^{\prime}$ will be the same as equation 1 with cTL(i,k-1) replaced by $c_{T_L^{\prime}}(i,k-1)$. Since $c_{T_L^{\prime}}(i,k-1) < c_{T_L}(i,k-1)$, $c_{T^{\prime}}(i,j) <
c_{T_{opt}}(i,j)$. However, this contradicts the original assumption that Topt is the optimal tree. Therefore, the subtrees must be optimal as well, and the problem exhibits optimal substructure.

(c)
Define a recursive solution for computing c(i,j) and write pseudocode for a divide-and-conquer algorithm implementing your solution.

Equation 1 already describes a recursive solution to the problem when we know the value of k. In general, we do not know the optimal key k to be the root, so, as in the matrix multiplication problem, we can try each possible k and retain the minimum cost choice. The only other aspect of the recursive solution is the stopping condition, which occurs when computing the cost c(i,j) when $i \not< j$. In this case, the cost for an empty tree or a tree containing one key is zero. Thus, the recursive solution is


\begin{displaymath}c(i,j) = \left\{ \begin{array}{ll}
\min_{i \leq k \leq j} (c...
... - f_k) & i < j \\
0 & \mbox{otherwise}
\end{array} \right. \end{displaymath}


 RECURSIVE-TREE(i,j,f) 
1 if $i \geq j$
2 then return 0
3 else $c = \infty$
4 for k = i to j
5 c1 = 0
6 for l = i to j
7 c1 = c1 + fl
8 c1 = c1 - fk
9 q = RECURSIVE-TREE(i,k-1) + RECURSIVE-TREE(k+1,j) + c1
10 if q < c
11 then c = q
12 return c

(d)
Give a recurrence T(n) for the running time of your recursive solution in part c, where n=j-i+1, and show that $T(n) = \Omega(2^n)$.

The recurrence for RECURSIVE-TREE is shown below (best, average and worst cases are all the same). Note the addition of the $\Theta(n)$term representing the computation of the added depth cost in lines 5-8.


\begin{displaymath}T(n) = \left\{ \begin{array}{ll}
\Theta(1) & n = 1 \\
\sum...
...{n} (T(k-1) + T(n-k) + \Theta(n)) & n > 1
\end{array} \right. \end{displaymath}

Using the substitution method, we can show $T(n) = \Omega(2^n) \geq c2^n$using the inductive hypothesis $T(k) \geq c2^{k}$ for k < n.

\begin{eqnarray*}T(n) & \geq & \sum_{k=1}^{n} (c2^{k-1} + c2^{n-k} + \Theta(n)) ...
...ta(n^2) \\
& \geq & c2^n - 2c + \Theta(n^2) \\
& \geq & c2^n
\end{eqnarray*}


The last inequality is true only if $\Theta(n^2) - 2c \geq 0$, or $c \leq \Theta(n^2)/2$, which is true for any constant c and sufficiently large n. Thus, $T(n) = \Omega(2^n)$.

(e)
Show that the recursive solution has overlapping subproblems and compute the number of unique subproblems in terms of n.

Below is a portion of the recursion tree for the computation of c(1,4)with overlapping subproblems indicated.

\psfig{figure=figures/dpga2.ps}

For each subproblem c(i,j), $i \leq j$. Thus, for a particular value of i, j can range from i to n for a total of n-i+1 values. Thus, for all values of i from 1 to n, we get the following number of unique subproblems.


\begin{eqnarray*}\sum_{i=1}^n (n-i+1) & = & \sum_{i=0}^{n-1} i + \sum_{i=1}^n 1 \\
& = & \frac{n(n-1)}{2} + n \\
& = & \frac{n(n+1)}{2}
\end{eqnarray*}


There are only a polynomial $\Theta(n^2)$ number of unique subproblems.

(f)
Write pseudocode for a O(n3) bottom-up, dynamic programming solution to the optimal binary search tree problem. Justify the O(n3)running time of your solution. Note that your solution must keep track of the root keys for each subtree and must include a procedure for actually constructing the tree.

Following the MATRIX-CHAIN-ORDER algorithm given in class, the following OPTIMAL-ROOTS(f) algorithm computes the optimal key roots for the entire tree and each of its subtrees given the key frequencies f.


 OPTIMAL-ROOTS(f) 
1 n = length(f)
2 allocate $c[0 \ldots n+1, 0 \ldots n+1]$
3 allocate $r[0 \ldots n+1, 0 \ldots n+1]$
4 allocate $sf[0 \ldots n+1, 0 \ldots n+1]$
5 for i = 0 to n+1
6 for j = 0 to n+1
7 sf[i,j] = 0
8 if i < j
9 then $c[i,j] = \infty$
10 else c[i,j] = 0
11 for ws = 2 to n
12 for i = 1 to n - ws + 1
13 j = i + ws - 1
14 for k = i to j
15 sf[i,j] = sf[i,k-1] + sf[k+1,j] + fk
16 q = c[i,k-1] + c[k+1,j] + sf[i,j] - fk
17 if q < c[i,j]
18 then c[i,j] = q
19 r[i,j] = k

Analysis and Explanation: Line 1 determines the number of keys n from the length of the frequency list f in $\Theta(1)$ time. Lines 2-4 each take $\Theta(n^2)$ time to allocate the arrays used by the procedure. Array c[i,j] will hold the optimal cost for a tree containing keys i to j. Array r[i,j] will hold the value of the key at the root of the optimal tree containing keys i to j. Array sf[i,j] will compute $\sum_{l=i}^{j} f_l$ in a bottom-up fashion to avoid making the algorithm O(n4) by computing this sum on demand. Lines 5-10 consist of two nested for loops initializing the arrays in $\Theta(n^2)$ time. Lines 11-19 follow the MATRIX-CHAIN-ORDER algorithm to determine the optimal tree costs and root keys using three nested for loops in worst-case running time of O(n3). The two differences worth noting are that kgoes from i to j instead of to j-1, and that sf[i,j] is calculated bottom up along with c[i,j] and r[i,j]. Thus, the total worst case running time of OPTIMAL-ROOTS is dominated by the nested for loops taking O(n3) time.

All that remains is an algorithm to generate the actual tree once the roots are determined. The following OPTIMAL-BST(r,i,j) algorithm will return a pointer to the optimal binary search tree for the keys i to j according to the optimal roots in r previously computed by calling OPTIMAL-ROOTS. The initial call would be OPTIMAL-BST(r,1,n).


 OPTIMAL-BST(r,i,j) 
x = ALLOCATE-BST-NODE()
k = r[i,j]
key(x) = k
left(x) = NIL
right(x) = NIL
if i < j
then if k > i
then left(x) = OPTIMAL-BST(r,i,k-1)
if k < j
then right(x) = OPTIMAL-BST(r,k+1,j)
return x

Since the analysis of this algorithm is similar to that of a binary search tree traversal, the running time is $\Theta(n)$. Thus, the running time for generating the optimal binary search tree is dominated by the OPTIMAL-ROOTS procedure, and is therefore O(n3).

2.
Suppose we want to design a greedy algorithm for the optimal binary search tree problem. Below are two possible greedy choices we could try. Unfortunately, neither satisfies the greedy choice property. For each, prove by counterexample that the greedy choice does not satisfy the greedy choice property.

(a)
Choose the middle ( $\lfloor n/2 \rfloor$) key as the root.

Consider a problem with n=3 keys with frequencies f1 = 0.1, f2 = 0.1 and f3 = 0.8. In the figure below the tree on the left will result from the greedy choice of taking the middle (second) key as the root with a cost of (0.1)(1) + (0.1)(0) + (0.8)(1) = 0.9. However, the optimal tree on the right has a better cost of (0.1)(2) + (0.1)(1) + (0.8)(0) = 0.3.


\begin{figure}
\begin{center}\psfig{figure=figures/dpga3.ps}\end{center}\end{figure}

Therefore, the greedy choice is not in the optimal solution and does not satisfy the greedy choice property.

(b)
Choose the highest frequency key as the root.

Consider a problem with n=4 keys with frequencies f1 = 0.1, f2 = 0.2, f3 = 0.3 and f4 = 0.4. In the figure below the tree on the left will result from the greedy choice of always taking the highest-frequency key as the root with a cost of (0.1)(3) + (0.2)(2) + (0.3)(1) + (0.4)(0) = 1.0. However, the optimal tree on the right has a better cost of (0.1)(2) + (0.2)(1) + (0.3)(0) + (0.4)(1) = 0.8.


\begin{figure}
\begin{center}\psfig{figure=figures/dpga4.ps}\end{center}\end{figure}

Therefore, the greedy choice is not in the optimal solution and does not satisfy the greedy choice property.


next up previous